What to do about Redundancy in the Vine?

I actually like the way digg.com handles this: when submitting a story, the system automatically searches for possible similar or duplicate stories. On Newsvine, this might be even better as we have the great virtue of tags.

Digg.com also allows users to "bury" stories and report them as lame, off-topic, duplicates or inaccurate. I would like to see stuff like this worked into Newsvine too. Right now there are three stories on the front page about the 1gb iPod nano.

It would be better if you could identify what it's a duplicate of, and after somehow vetting that (x number of duplicate "votes" or something), "collapse" the stories into one.

Also, I've only posted one link so far, but there did appear to be some kind of "this appears to be similar" message, although in my case it was for a completely unrelated story.

Another problem that I see already here, and that made me stop reading digg, is the continual posting of "old stuff". Not necessarily duplicates, but people finding things on the web that they think are interesting and not realizing they're ancient history. This was particularly a problem for digg because most of the users seemed to be children. One example I remember is when the "IBM mouse ball memo" made it to the front page. I saw something similar to that on Newsvine yesterday, but I can't find the example now. It may be a harder problem to solve, especially if it's not really a problem for everyone.. if you haven't seen the dancing baby, it may be news to you.

I agree with Smithco. Just because there are three "redundant" stories on the vine does not mean that all three stories say the same exact things. Multiple sources are a good thing.

I agree that multiple sources are a good thing, but there really needs to be a way to weed out the duplicate articles. Especially if they are AP stories, and were just submitted from different news sites.

I especially like the way Digg handles it. It searches the database for possible dupes before posting.. Of course, not that anyone over at Digg actually cares about posting dupes...

I think Christopher Masto, smithco and wolfger all make some very good points that are linked well. If Newsvine could find an algorithm (or even a functionality button that would allow users to manually make the change) which allows mulitple stories to be combined into one, each accredited to their respective seeders/authors then the stories are more valid overall. To repeat wolfger; multiple sources are a good thing.

Additionally (I wish you could edit comments in the first 5 minutes after posting) you have the bonus that once submitted the alternate sources would be red flagged by the existing architecture of Newsvine; thus disallowing any future seeding of that article.

This is from the Company Info page

Established news sites like MSNBC and CNN are able to bring you a wealth of information from around the world mostly by editing and syndicating thousands of Associated Press articles every day. Depending on what your favorite news site is, somewhere between 70 and 90% come from the A.P. Newsvine syndicates these exact same articles but takes things a step further.

Where its says "...Newsvine syndicates these exact same articles..." does that mean Newsvine itself has these stories already loaded or the members of Newsvine syndicate these stories through seeding. Because wouldn't that be causing some redundancy too.

I'm not sure if you guys noticed this, but recently Newsvine changed it's reporting verbiage to reflect new "Report As ..." labels. One of the new labels is "Duplicate Story".

Simply click on the "!" icon that appears below each article/seed title to report the item.

Also, concerning Alexander's first comment about automatic notification of an already seeded story: It's already implemented. When you click on the "Seed to Newsvine" button the popup window masthead will be red if the story/article has already been seeded. Additionally a link to the seed will be provided so you may join the discussion about it!

^^^Sorry for my incredibly dumb post up there (number 2). This is my first day here and i barely checked out the stuff on newsvine before i submitted that comment.
I also noticed Newsvine has no edit button eheheheheh....

@kilometer Does this really stand up if the links are added to the first seeded source of the article? Say you submitted an article from CNN which I then found on BBC; would me adding the link directly to the bottom of your post for further reading/discussion/information be a bad thing? It would not be cluttering but would offer more sources for (possibly) further information, confirmation and different takes on the topic.

I think this would work, but I am asking if you foresee something which I do not.

The only problem i see with reporting duplicate seeds is that sometimes both dupes get comments -- there needs to be some system for integrating these comments. other than that, I strongly support user-reporting of dupes.

It is interesting to see different angles on the same issue, but redundancy is a definite put-off. What'd be good is for articles on the same subject to be "clumped" together, so I see links to all related articles about a given news item. Based on reviews, posts that are too similar can be merged. This would make the process a tad more complex, but it's a not an easy problem to solve anyway.

I think that rather than search by possible keywords, the way Digg does it, just search by root URL. I look forward to whatever the NV team has cooked up.

I think that since people want newvine to be "full of info" i.e. few redundancies three solutions might help:-
1. adding a 10 word summary - this will have the key words, so that detecting similar articles will be easy.
[Thsi might be like the META tags to be used by newsvine to help collate similiar articles.]
2. having the system find you the entries with similar keywords.
This way, you will not accidentally start a similar thread.
3. News is topical and "perishable" so tagging the keywords with a date (e.g. 10 days old) will allow you to match common articles. [This is a suggestion to newsvine - I really don't know how to combine the time aspect
- but I suspect it will improve the relevancy.]

How bout an x that clears the story from you own home page. This way u can track what stories you have read or don't care to read.

I like the idea of clumping related stories together. Maybe having one main article, with a list of several other articles below. Kind of like how Wikipedia has cites and links to additional materials at the bottom. Also, that way the discussion would all be in one central location.

Can't wait for the new "industrial-strength-redundancy-and-relatedness-engine" coming this week! Sounds great.

I'd prefer that exact dupes be deleted; maybe their comments could be ported over to the first instance of that article.

Dupes that have a slightly different angle or are reported by a different source...I'd prefer these to be grouped like Google. I like having the option to click on different sources and get different points of view.

It's funny that I read this article because I have one that appears underneath it if you filter for the newsvine tag. I saw your article, but the only reason I still typed mine is beacuse I commented about the roles of bloggers/social journalists and internet information efficiency in general and less about site features. Though seeing the two next to eachother is funny to look at.

like someone who seeded a story in december and then the same story was seeded in february again -_-

Mike D.:
Can you title it: Doppelganger! The industrial strength redundancy and repetitiveness engine.

Ive noticed that a lot of the AP articles seem to be duplicates as well - but there's no way of warning of this - does that mean they have priority over user articles?
Will the new system allow the people who are posting these see if they are posting something which a newsvine user has already managed to post about.

dave proth writes:

How bout an x that clears the story from you own home page. This way u can track what stories you have read or don't care to read.

Thats a great idea!