New Rating System?

I got it! I got it!

No, wait, I haven't…

mh…

I mean, there's pi, but where's the apples in this?


Pff…academics…

 

The rating system works basically like this:

Something that gets one good vote is liked by a few people. However, if something gets a lot of good votes, it must be good, but also it must be applicable to a lot more people, so the rating goes up. It's almost more of how well known and respected an ensemble is more than anything else.

It is weird... and I don't think its useful because something can be great but just new, so it'll score low... personally I feel its less useful than the user voting. Ah well.

 

Now that I understand the new voting system a bit better I think it's a step in the right direction!

 

i still don't like the voting system. some months ago i
decided for myself to completely ignore the votings
and i don't think that i will change my mind with the
new system. but well, thats just me.

 

Yeh, me too. It's the comments that count. and I still vote 10 for anything I like.......... never 6 or 8 or 2.

-Only anal types grade the ensembles.

 

Hmmm, I guess it does make more sense now. The scale is somewhat brutal, though... some builders might feel a bit hard done by. ;-)

 

Yup, I in fact do wonder a bit about the new rating system - what kind of maths is used to calculate the new rating? My ensemble "Laserbrew" is still at a nice position in the top 100 (at #6), has a user rating of 10 (formerly something like 9.65), 24 user votes, 3440 downloads and has been uploaded 2003-07-24 - but the new rating says just 7.32. All values except the number of votes are comparable to LIME.lite, but the new rating makes it a huge difference of 2.68/10 - funny system. ;-)

I´m just quite curious how it will rate new ensembles...if you still can find them, it´s ok (although NOT intuitive).

Best,

Tim

 

rachMiel's soap box

rick mounting soap box ...

the entire ratings system, old or new, does more harm than good, in that it narrows, rather than expands, users' musical palettes by affecting, consciously or subliminally, users' openness to uploaded ensembles.

it would be a whole different ballgame if ensembles were not rated. instead of thinking ... hmmm, this got a 9.5, so it must be good, even if my ears tell me different ... or, this got a 5, i won't even bother downloading it ... you would think ... hmmm, looks interesting, let's see if this baby speaks to me.

user comments: good. user ratings: BAD!

rick demounting ...

 

I've never checked the score before downloading...or checked back on my own 'results'. The votes have never infuenced me or impressed me.

However, the words 'drum machine' 'beat box' 'FM synth' will often act as a deterant for downloading.

;-)

 

trying to pull the wool over other people's eyes?

sowari

 

i agree with you here tim (although LIME.lite has had 85 votes)
2.68 is a big difference, if lime.lite has a rating of 10 your ensemble should have a rating of 9.2 or something like that.

sowari

 

Re: rachMiel's soap box

I agree. Especially with the dastardly 1-er on the prowl!

 

Well here's my own last word on the subject. I totally agree with Rachmiel. Recently I have been working on niche ensembles that are really useful to a small number of people. With the previous system, I could easily see that a small number of people really liked my work, because it showed both downloads and user rating. Now it just looks like things are either really good or really bad.

A good example is Rachmiel's own monster ensembles, which are brilliant if you want that type of thing, but won't get a large number of people appeciiating that type of thing.

Also, the number of downloads are corrupted. There have been several times when the NI website was down, and things got several thousand downloads while people were trying to get things, but couldn't open the files. All those ensembles now have really good ratings.

Recently I posted a MIDI patch bay, which is totally unique. There isn't anything else like it in the library at all. I felt happy that it would receive good ratings only by people that wanted it, and it would end up easy to find on the top lists. I even thought maybe NI would want to include some of my work as premium ensembles in their own products. Now I just got buried somehwhere with a low rating, and probably that's the last I'll ever hear about it.

Anyway, I guess this is pretty well goodbye from me, at least for a while. Sorry to say, but duty calls elsewhere. It's been fun chatting with you folks. And I've learned valuable lessons, like for example, if you use software from companies that don't maintain "known issues" lists, you'll spend mostr of your life fighting with bugs.

Happy sailing, folks.

 

Ernest's Kafka filter has a '10' voting from seven users but it still only has a 3.67 rating. I'd say that's rather unfair...

The new rating system is clearly having a negative effect.

 

that is because, the "rating" system is also based on numbers of downloads.


BUT i agree with you.

 

I'm not shure about "negative effects" of the new rating system. I have to admit that - as a new Reaktor user - my orientation at first went along the ratings: The first ensembles I've downloaded were ones with high ratings. But soon my criteria changed: By reading the comments, visiting this forum, looking at the explanations at reaktions.com I stumbled across names of obviously interesting ensembles or creators; by trying the ensembles I realized what's interesting for me; and I began to dig the library according to these lines, ignoring the rating more and more. This way it's easy to find, for example, everything by ernest - his ensembles are connected to the "top" of the library by far more "associations" than only the ratings. Perhaps the ratings alone are less useful than before - but because of the different sorting possibilities they're not completely useless. No reason to despair! ;-)

 

I'm not despairing at all. ;-) By negative effects I mean demotivation of builders. Ernest's reaction illustrates this perfectly.

Don't forget the builders make and upload these things for nothing. It's obvious they'd like as much positive feedback as possible from the community. 'Low' ratings don't achieve this.

 

*sigh* Here we go again. I think this whole discussion is a good example of the old "you can't please everyone" axiom.

I'd invite people to read some of the comments made when the last rating system came out in August 2003;
http://www.nativeinstruments.de/forum/showthread.php?s=&threadid=7170

At lot of these changes are things that people asked for...
I think making it difficult for a brand new instrument to be a top-100 is good.

I think not being able to vote unless you've done a download is good.

The biggest problem I see right now is that there's too much weight put on the number of downloads, especially when the download count used for the current rating numbers does not take prior outdated versions into count. I can tell this easily from the ratings given to some off my ensembles that were recently updated and so have relatively download counts.
(maybe Markus was saying that won't be the case for future updated releases of ensembles, but it is definitely the case now that only the most current release's download count was used in coming up with the numbers).

I've said this before and I'll say it again. If I upload something like the EventWatcher - that's only going to be used by maybe 5% of the Reaktor world - I don't think it should have a high *overall* rating. Regardless of how useful it is to that 5%. Someone new to the software who wants to see what kind of cool instruments there are would find EventWatcher completely useless. They should have to go looking for it, and I figure they will when they get into it enough.

Saying that there's a problem with the fact that things that are quirky or aren't going to be enjoyed by a lot of people don't have high ratings is like saying that Billboard shouldn't publish it's charts because it skews uneducated music buyers into only buying vapid top-40 hits, and never exposing themselves to really artistic music. Come on, that's the whole point - to help people who don't know what they want make up their minds. People who are into the software and into the sounds, or into the abilities of this builder or that builder, don't even pay attention to the ratings. They dig into the library and find interesting things that suit them. They become fansd of particular builders.

As the post I linked to says, I think there should be ratings for different categories (technical, looks, usability, sound quality), but then everything in the current library would need to be reset to zero and voting would have to start over.

Another thing I think would be really helpful would be a "builders I admire" list associated with each person's profile. A way for each person to list their top-ten builders with maybe a comment about each one. Like the lists on Amazon. You should be able to link to te builder's "admire list" right from each of their instruments.
That way if you find a builder who's instrument you like, you can start linking to other builders to find things that might be similar (musically or technically).

So...
...do I think these changes are ideal? No. Do I think they are an attempt to address issues that we ourselves raised 10 months ago? Yes.
Do I think people would bitch if the rating system were taken away all together? Yes. Do I think they'll bitch the next time it's changed - no matter how it's changed? Yes.

- CList

 

Ernest, the stuff you (and number of well-known other contributors) upload is of such interest to users, that I think no matter how nuts or ill-devised the voting system is, savvy people will simply search against the creator's name for any of their other work ... so nothing will ever get disregarded because it is not at the top of the charts for those who make the effort to seek out the nuggets.

I don't think the voting system ever was a good measure for finding interesting contributions ... it is impossible to properly compare one ensemble against another and give them ranking numbers.

 

Originally posted by CList

*sigh* Here we go again. I think this whole discussion is a good example of the old "you can't please everyone" axiom.

It's true.

I think making it difficult for a brand new instrument to be a top-100 is good.

I would've preferred doing it by requiring top-100 ensembles to have more than a set number of votes. This is a lot more straightforward than some creepy votes * downloads algorithm. It prevents the 1-vote 10s from being on top all the time, and it doesn't penalize something just because it hasn't been around a year.

I think not being able to vote unless you've done a download is good.

Agreed.

The biggest problem I see right now is that there's too much weight put on the number of downloads, especially when the download count used for the current rating numbers does not take prior outdated versions into count.

In my opinion, putting -any- weight on the number of downloads is silly. Certain uploads will be attractive to more users than others. However, even if an upload only attracts a small number of users, it may be exactly what those users wanted. The current system is also useless to things like macros, which tend not to get lots of downloads. This wouldn't be a big problem, but when you search for things, it shows the creepy rating, not the user rating. What the users say is masked almost entirely by the number of downloads. This makes the ratings even more useless than they were before.

I've said this before and I'll say it again. If I upload something like the EventWatcher - that's only going to be used by maybe 5% of the Reaktor world - I don't think it should have a high *overall* rating. Regardless of how useful it is to that 5%.

I completely disagree. Something with a low number of total downloads so far may be useful to a lot of people.

Saying that there's a problem with the fact that things that are quirky or aren't going to be enjoyed by a lot of people don't have high ratings is like saying that Billboard shouldn't publish it's charts because it skews uneducated music buyers into only buying vapid top-40 hits, and never exposing themselves to really artistic music. Come on, that's the whole point - to help people who don't know what they want make up their minds.

I think we should strive to be a bit more trusting of fellow Reaktor users than Billboard is of 14 year old girls!

People who are into the software and into the sounds, or into the abilities of this builder or that builder, don't even pay attention to the ratings.

I know a lot of the builders do though, and it can be a bit annoying when you work a long time and someone rates you a 1. People have complained about this a lot. I am not just against the new rating system. I am against any rating system.

They dig into the library and find interesting things that suit them. They become fansd of particular builders.

Does that mean I have fans? ;-)

As the post I linked to says, I think there should be ratings for different categories (technical, looks, usability, sound quality), but then everything in the current library would need to be reset to zero and voting would have to start over.

Ratings are relatively new anyway. I have no issue with resetting the ratings if a significantly better system would take its place.

Another thing I think would be really helpful would be a "builders I admire" list associated with each person's profile. That way if you find a builder who's instrument you like, you can start linking to other builders to find things that might be similar (musically or technically).

Not a bad idea, but that limits its usefulness to forum viewers only. I suspect most people going to the library don't frequent the forum. I think the forum is skewed towards oddball Reaktor users anyway.

So... do I think these changes are ideal? No.

Sing it sister.

Do I think they are an attempt to address issues that we ourselves raised 10 months ago? Yes.

Not my issues. ;-)

Do I think people would bitch if the rating system were taken away all together? Yes.

People will bitch if Kerry wins over Bush. It's about doing what's right... damnit! *pounds fist*

]Do I think they'll bitch the next time it's changed - no matter how it's changed? Yes.

I should hope so. Let's get rid of that silly thing. It smacks of commercialism and hand-holding for Sessions users. Let them be bewildered for a bit. The additional exploring will be good for them. No one ever learned anything by searching for "drum machine". Which, by the way, now shows anything with the words "drum" or "machine". I think all things should be joined via a boolean AND by default.