Expert Curators, WisdomCards & The True Wisdom of @organizedwisdom

9112010

Note added 2010-11-12: Anyone who wishes to can now email info@organizedwisdom.com to let his/her profile as expert curator removed from the site of Organized Wisdom (see comments by Unity Stoakes, Co-founder OrganizedWisdom)

————————————————————————————————-

Twitter and other Social Media can be full of random rubbish, but can also contain useful information. Personally, I use Twitter for work-related tweets about 95 percent of the time, and I choose the people I follow carefully so that I’m not overwhelmed by a flood of tweets. As I’ve said before: people who I follow are my human filter to the Twitter Noise. And I hope that is vice versa.

Organized Wisdom has created a new way for people to use the internet to solve their health problems by using experts. We think experts are the missing gap between two worlds: heavily simplified health encyclopedias & dictionaries and complex stuff on the internet (where you are quickly overwhelmed by all the health stuff).

OrganizedWisdom® uses WisdomCards™ , a unique service which helps people easily discover great links, curated by experts and organized by topic.

So far so good. Besides being a creator of content (which is nothing more than being a source for automatically created links present in my health-care related tweets), I’m also a “consumer”. Because I’m interested in Endocrinology I follow @EndocrinologyOW on Twitter. There are many other OW topics on Twitter, that you can follow.

This is why I got this tweet in my twitter stream (I don’t follow @ActivityDigest, but I see its tweets once it mentions me).

I felt flattered: “recommended as an expert curator for Organized Wisdom“…. That sounds like any scientists could only dream of.

And before I realised it, I retweeted the message (repeated the message to my twitter stream).

Immediately I felt a bit uncomfortable. And this feeling grew as I saw almost everybody in my Twitter stream being labeled as an “Expert Curator”. … and everybody retweeting his/her “pat on the back” and congratulating each other….

I don’t have the original retweets, but a search for “Expert Curator” shows that last weeks “expert-curator”-retweets still continue

Note (1) the real surprise of some people (Wow, I never heard of Expert Curator for quality links before) and (2) the chaff (endless beauty) between the wheat ((BMJ_Group, American_Heart). I mean: if some of these accounts would start following me, I would probably block them and report them for spam! Really!

Activity Digest continues its activities. Now everybody knows he or she is a curator, it tweets what the curators have achieved this week. Probably OW is hoping curators retweet their achievements and spread the word for them. We are not only used as living filters, but even more so as “living ads”!

It made me frown, but the worst thing that struck me today is that the twittered information isn’t really filtered for quality. About a quarter is what I would call SCAM or at least CRAP.

Part of the problem may be that Organized Wisdom doesn’t only share links from “Health Centers”, but also from Wellness Centers (Aging, Diet, Exercise & Fitness etc) and Living Centers (Beauty, Cooking, Environment). Apparently one card can have information for 2 or 3 centers (diabetes and multivitamins for example)

I feel used.

Organized Wisdom uses the credibility of me and other curators, including so-called “top expert curators” as Dr Pho (Kevin MD – blog), to cover up the incredibility of others, with the intention to lure users in.

Who doesn’t believe top curators like the ones below? Who doesn’t want to be considered one?

Going through the “expert curator” Twitter search, I saw (only) one very sensible man, EdBennett, who manages web sites for a large academic medical center and creates overviews of social media usage by hospitals. Friendly, he asked to be removed from the Expert Curator list.

I think I will try the same, although (considering the answer of dr Ves), that may not be a sinecure.

Perhaps we can try a concerted action. You know, the power of the crowd….

I am glad you brought up the “written by people who do not have a clue, let alone ANY medical training [of any kind] at all.” I have no experience with any kind of medical education, knowledge or even the slightest clue of a tenth of the topics covered on OW, yet for some reason they tried to recruit me to review cards there!?! That was my first impression of the company, if they have a bunch of people like me as the final word on their cards, the info will be so off base the site will give the big fail so quick. In my opinion, medical knowledge or advice is not something I would trust from any random joe, so why would I trust anything from random people who work there, considering they have absolutely no experience or knowledge? They advertise that a doctor reviews every card, personally I think that is a bunch of BS, considering neither me nor others that I know work higher up there have any training whatsoever, guess anyone can be a doctor now…

And Josh,

You are correct about some health sites’ contents written by people with limited or non-existent medical knowledge or training; the public at large do not know this. In fact, one of my neighbors, a “MA” or “Medical Assistant,” writes for OrganizedWisdom.com.

17 responses

Thank you for your well articulated post and constructive feedback about what we are building at OrganizedWisdom. You highlighted a few of the issues that we are working hard to improve, but your feedback has also pointed out some additional things we now know we need to take a closer look at so for that I’d like to say thank you.

As co-founder of OrganizedWisdom, I take each issue about our service very seriously and wherever possible immediately work to fix them. In fact, everyone on our team does. So I can assure you that we will look into every concern you raised and hopefully start a dialog with you online or off.

First, let me try to explain what we are trying to do. It’s a huge goal, and it isn’t an easy one to achieve. And most certainly, we won’t be able to do it without the support of people like yourself or those with expertise in researching and searching the Web better than most people.

We are trying to build a better way to guide people to trusted health resources. We believe we can do this by making sure people (not just machines) are part of the process. We are creating a system that filters and organizes the public contributions such as link sharing of a select group of people that have identified as experts in a particular health related area. We are trying to crack this code because we believe that by organizing the collective wisdom of doctors, nurses, health researchers, experienced bloggers, librarians, patient advocates, etc. and filtering and structuring this knowledge in a way that ordinary people can get access to it for free, we can help a lot of people. We will be able to cut out the majority of the bad content. (And when we figure out a way to cut it all out we will jump for joy!).

This type of big mission is not easy. It takes an extraordinary amount of hard work and resources. Most importantly, it takes people who are willing to share their knowledge and experience for the benefit of others. You are already doing it now on Twitter. We just want to organize it so people (including yourself) can actually use and find the resources you are sharing. We do this by tagging it, filtering it, applying taxonomies to it and so on. It is very hard work and is not yet perfect. We’ve been at it for nearly 4 years and we feel we are only 10% there. But intuitively we know that it doesn’t make sense to leave something as important as ones health up to a computer alone. We want to make sure that the wisdom from the world’s best experts gets baked into the algorithms for the first time…this we know will be useful to a lot of people.

We are in the process of building some amazing new innovations to improve what you now see. We hope you will continue to share your feedback, let us know what we can do better, and also let us know how we should be improving the service to help you and those you care about.

I do know what you are trying to achieve and what your aims are. That’s why I joined a while ago and followed you too.

You started by saying:

You highlighted a few of the issues that we are working hard to improve, but your feedback has also pointed out some additional things we now know we need to take a closer look at so for that I’d like to say thank you.

However you never detailed what you would like to improve and the areas you want to take a closer look at”.

I can see various things you could work at.

1 Topics.
-Most have to do with lifestyle and natural health. Of course of interest to many, but since there are many contradictions in this field it is even more important that the information is accurate (and without any smell of partiality/commercial input).
– Some of the topics are not without controversy. Take homeopathy and chiropraxis

2. Experts. As said there are many among the “experts” that I wouldn’t characterize as such. Even a doctors title doesn’t guarantee quality.
And if a topic is controversial, what about the expert? An expert in homeopathy who only gives (positive) homeopathy-related links doesn’t lend credibility to homeopathy and “misinforms the public”. The same is true for anti-aging cremes etc. What a rubbish!

3. Links are shown out of their context. For instance a Cancer and Homeopathy WisdomCard™ shows Expert Curator Liz Ditz who is sharing a link about Cancer and Homeopathy. The link she shares is a -I would call it- dangerous article by a Dr who is working in an Homeopathic General Hospital, in India “reporting” several cases of miraculous cures by Conium 1M, Thuja 50M and other watery-dilutions. I’m sure that Liz Ditz, didn’t say anything positive about the “article”. Still it seems she “backs it up”. Perhaps she tweeted: “Look what a dangerous crap.”

Thanks for more great feedback. Let me try to address with some more specifics. We are also updating our own FAQs and will work on a blog post on our site to make sure this important feedback reaches everyone who may have similar concerns or feedback.

1. Control of your Profile:

First, we are in the process of building a new profile system that allows you to claim, own and control your profile in a much more sophisticated manner. We had hoped we could rely on Twitter oauth for this and each person would simply be able to control their profile from their Twitter account. We were wrong. So for the past few weeks we have been working around the clock to build a more controllable profile that you can claim and control.

To be clear, you can now control things like your bio and the links directly from Twitter, but there is a lag in data syncing that we do not believe is acceptable.

2. Deleting Profiles:

Anyone can now email us at info@organizedwisdom.com to have their profile removed and we will do so although our hope is that as we create enough value with our system, just the opposite will be the case. Our FAQs are being updated now to better communicate this policy.

This is an issue of debate, and we are open to suggestions from the community long term here. We feel that it is appropriate to delete accounts at present if people ask because we do not yet have the features for you to control your own profile properly. However, there is debate about this issue. We are not only creating a service which will give experts useful tools and analytics about their contributions, but to be a service that will guide ordinary people who need help to trusted experts. One day we hope to validate and measure ALL health experts online so that people have a transparent way to see who is credible and who is not. Imagine, if this system let all of the non-credible quacks opt-out rather than serve as a transparent record measuring every health experts online contributions and identifying quality and influence. Imagine if Ebay let bad sellers opt out of their profiles if they had negative feedback. Thoughts?

3. New Features Designed for Experts

We are excited about what’s to come. I can’t share all of the details, but we are building a number of tools to help the Expert Community have more control over what they are associated with using new tools to create, edit, organize, add to, delete, etc. WisdomCards…we are also making improvements to the Activity Digest giving you more valuable insight to your contributions and how they may be helping others.

4. We have stopped the ActivityDigest tweets:

Our intent was to communicate to everyone that they had this new Activity Digest. A vocal few don’t like this so we have turned it off. We are open to better suggestions for how we should be letting people know about this new feature.

5. Lifestyle and other Health Experts:

This is another challenging topic. We are building a comprehensive digital mapping of everyone who is active and contributes online related to health issues. This includes areas of health that are related to wellness, complimentary medicine, living issues, etc. Our system gets more intelligent over time. It starts to identify who is legit, who is not, etc. This is one way we are leveraging technology to help us by building out a quality score called our WisdomScore which is still in alpha. Over time our system begins to recognize whose contributions are not valuable or helpful. Because these scores are all transparent people will be able to clearly see which links, experts, content is deemed more credible than others. We plan to filter out the junk of course.

6. Addressing Out of Context Links:

We make daily improvements to our taxonomy, our filters, deleting bad resources, bad curators, etc. We are doing amazing work on this issue with better natural language processing, changes to our technology, and by analyzing patterns over time. We still believe the approach of integrating expert curated resources in with great technology is the way to go. It’s getting much better everyday although we realize not yet perfected.

7. Communicating with Experts:

We are working on building an Expert Advisory Board, who can serve as an independent sounding board for us and help us shape the long term solutions to a very big challenge. If you or people you know would be interested in this please email me at unity @ organizedwisdom.com. We are also doing our best to reach out one on one and via our blog and resources like our FAQs. I welcome any opportunity to dialog with those who have ideas about what we could be doing better. As I said before, we are very much just beginning.

I hope I addressed all the big issues you raised with more specifics. Thanks again.

The only thing that alarms me about OrganizedWisdom is that they’ve made me an expert curator 😉

Jacqueline, I think your points are very well made. There may well be a good idea in OrganizedWisdom trying to get out, and I appreciate Unity’s responses.

However, the quackery that you identify foregrounds the fact that until OW applies stronger filters and takes a long, cold look at both the subjects curated and the curators it elects (shouldn’t the community be doing this?), it may neither serve its users in the way in wants to (I’ll confess to still being somewhat vague as to its mission), nor receive the widespread use we assume it aspires to.

OW remains an interesting work in progress.

An acid test as to its current utility may lie within a self-reflexive useage survey among its own curators. Personally, I don’t use OW; maybe other curators do.

1) The filters you mentioned are critical and is something we do on an ongoing basis using human reviewers as well as technology. We have an active review process, but alas some bad apples slip through, so it is an ongoing battle. Additionally, we have just reviewed our policies and process and are revetting all of the curators in our system to purge any curators that we feel don’t meet these quality standards.

Regarding this, there are big issues that cause debate among the health expert community. For example, what types of experts should be considered and included? We had someone opt-out today because they believe we shouldn’t include anyone who promotes complimentary or alternative medicine. We disagree and believe patients should have access to multiple viewpoints as long as they are not what most everyone would consider to be quackery and so on. We address this issue by transparently labeling who is what type of expert based on their activities and contributions. That way the patient can decide if they want to see resources that have been shared by doctors, or nurses, or alternative specialists, or patient advocates, and so on.

2) You are correct in saying the community should and can help with this. First we are in process of building a variety of tools to do this much more effectively. Second, there are several ways this is already happening now. Experts are actively recommending and voting up good quality links, they are alerting us to bad profiles or activity, and so forth. What we are in process of building is much more interesting however and we look forward to having people like you participate if they want to help.

3) Your idea for survey is great so we are going to implement it soon! We’ve done one on one reach outs but I think a community wide survey will be incredibly useful.

I appreciate your openness. I also appreciate the fact that you’re working on it and already made some changes: stopping the ActivityDigest tweets & directly removing profiles upon requests.

However, I do agree with Andrew’s comments about the quackery.

Quackery or nonsense are so inherent to many topics you have chosen in the field of wellness, complimentary medicine etc. Many among the so called experts provide information that is even less useful than the free local papers with dozens of paper coupons that flood my mailbox at home.

You say that “We had someone opt-out today because they believe we shouldn’t include anyone who promotes complimentary or alternative medicine.”

It might be Liz Ditz, at least I know she opted out after I told her about her wisdom card (mentioned in the example above).

She told me “AIEEEE…. didn’t sign up with Organized Wisdom that I know of” She felt she was used for credulous support for homeopathy & naturopathy, because she tweets a lot about CAM. Now there are only quacks as experts left.

You say: “We disagree and believe patients should have access to multiple viewpoints as long as they are not what most everyone would consider to be quackery and so on”.

In principle, discussions and access to mulitple viewpoints is ok, except when it is complete nonsense. Furthermore, in this case, Liz was the only person that I would regard as “reliable”, but the link that you put on her wisdom card led –as I said- to a nonsense article about homeopathic medicine (http://www.homeopathyworldcommunity.com/forum/topics/more-cancer-cases-homeopathy?xg_source=facebookshare) that she would never approve of. She must have said something negative about it, but that nuance has gone. It was only the link to the paper that was on Liz’Card, not what she had said about it. Without the context, information can become misinformation.

The example has gone, the principle hasn’t.

It is very fundamental. You can make the most fabulous tweaks and amazing filters, but it can never work without manual checking (at least with respect to the tricky topics). Furthermore, as an organization, you should define beforehand what is “credible” and what not.

Personally I don’t belief in automatically created wisdom cards. And I don’t think people have the time to check the credibility afterwards.

As it is right now, I would change their name in “stupidity cards”. I’m sorry, but I have looked around quite a bit, and I have found no information that was useful to me (that I didn’t pick up in my normal twitter stream). I rather have a well designed and checked (mini-) wiki.

Furthermore, I still believe that truly reliable and expert people on Twitter are “misused” because they give credibility to the non-credulous people. Often they don’t know that they are used and how they are used.

See for example the Asthma Health Center.

Dr Ves is a really an expert on this topic. But he is beaten by Allison Harkett, by number of links that is, but still…. Allison has “Diabetes, Asthma and a few more problems have meant that I have had to learn how to keep myself slim, fit and healthy.” And has the website slim-fit-health.info/newsletters/fat-loss-now/.

So there you go.

Of course a separate category is Asthma Alternative Treatments WisdomCards™, with a high score for Acai Berry for Allergic Asthma. This is clearly a link-bait story…. (I’m not going to link to it)

You can remove this as a source, but we still have many others.

We have cosmetic surgeons (not the Ramona Bates types -these are excellent), we have homeopathy, chiropraxis, herbs, slimming products etcetera.

I’ve thought about it, but I think it will never gonna work as a credible source.

@Heidi – Fancying seeing you here 😀 I think you delineate the ‘on the one hand’ position perfectly.

@Jacqueline – I really appreciate your having furnished Unity with such a crushing example of why OW is failing at the moment. That @DrVes is being trumped is not a victory for people seeking information about asthma.

@Unity – This isn’t a triumphalist clarion call for the primacy of the healthcare professional’s opinion, and a concurrent belittling of the patient’s opinion on my part. You follow me, and will therefore know that I advocate transparency and democracy in the health conversation, and support the epatient movement.

However, I value accuracy and relevance above all other things. Anything that diminishes the likelihood of people seeking health information encountering the most reliable, relevant, timely and accessible data *first* is failing.

Admittedly, that includes just about every source on the web at present :). But IMO, at the moment OW is exacerbating the problem rather than helping to solve it.

I value accuracy and relevance above all other things, too. And looking at the site more closely I come to the conclusion that given the current basis they will not succeed. I also wonder where they get their money from.

I found the example of Liz even more striking. An anti-quack was “used” to give credibility to a paper describing the usefulness of homeopathic treatment in cancer!!! (it was her link, but she probably said no positive things about it)

@all I Finally made up my mind to let my profile removed from organized wisdom.

@andrew thanks for your insights. we take them to heart and action, and certainly agree that reliable, timely, and relevant information is key. As you well know, these transformative ideas (mixing experts into the process now controlled by machines) is a significant undertaking. It isn’t a failed concept, it is only a work in progress and one we know we will solve in the hopes of helping a lot of people.

@heidi anyone can request to have their profile deleted which has always been the case.

@jacqueline and all, I just wanted to share an update so you hopefully know how much we respect and appreciate your feedback. For the 4 of you who have opted out, we hope you will come back soon.

Here are just a handful of the changes that have been made or are underway.

– We are in process of updating a significantly more stringent curator review standard and will be publishing by end of next week
– All 5,000 profiles in our system are being re-reviewed and re-verified (and removed) based on updated standards within the next 7 days. We expect this to make a dramatic improvement system wide.
– We are in process of segmenting or removing all non-health related curators which we think will improve quality greatly as well. We are especially focused on the complimentary, wellness and lifestyle areas.
– Over the next few months, we are building the next version of the Profile and Activity Digest that you will be able to claim and control directly on our site instead of just relying on Twitter. This will provide for much more direct community involvement and control.
– We are in process of making significant changes to our technology to improve the accuracy and relevancy of how links get organized, and we are building easy to use tools to give curators ability to control this if they’d like. These features will be rolling out early next year.
– We are changing WisdomCards to more clearly label what type of curator someone is so people can easily distinguish between a doctor, a patient advocate, a health blogger, a nurse, a researcher, and so on. (This is already in place, we are just making it more apparent for visitors).
– We are publishing clear definitions of specific terminology like experts and curators
– We have updated our FAQs and site information to make it easy for anyone to have profile deleted
– We have deleted profiles for the 4 people who have requested to be removed within hours of asking
– We are in process of organizing an Advisory Council made up of doctors, patient advocates, researchers, and people who care about trying to innovate so that we can get continuous feedback on our longer term product plans
– We have responded openly to the critiques and more importantly altered our development roadmap to improve the quality issues requested.
– We have done this because we truly care about developing a quality service, and we care about building something that helps people.

You probably won’t see me comment again on these issues only because we are busy making improvements. Please do check in again with an open mind in the future and we hope you understand it’s a work in progress. What we are building isn’t easy and we appreciate all the support we can get so that we can ultimately build something that will help a lot of people.

[…] worse, blog posts are sometimes mentioned to give credit to disputable context. I’ve mentioned the tactics of Organized Wisdom before. More recently a site called deathbyvaccination.com links out of context to one of my blog […]

Recently many of my physician colleagues had their own websites and blogs plagiarized and copied by OW’s new schemes. We’re talking about health information for patients. At this point, just in 6 days I spoke to 80 physicians who did not even know they had profiles on OrganizedWisdom and yet they were participating in this scheme. There are now over 5700 health care “curators” on OrganizedWisdom, but yet it seems not many know about it.

I certainly consider the founders and investors of OrganizedWisdom as geniuses for putting together such an elaborate website which at one point must have been a great company with amazing potential. But crossing ethical, moral, and legal boundaries … that goes just too far to ignore.

If you’re on Twitter, voice your opinions on #EndToFarms – I created this Hashtag to help expose such practices by unethical healthIT companies.