Towards a reputation system suitable for SteemSTEM
Although several reputation systems for Steem have been proposed (some of them even very recently), I believe that none of them are appropriate for implementation as a reputation metric that could be potentially used on our up-coming SteemSTEM app (to be officially released very soon).
[Credits: @hightouch]
Therefore, I decided to build one myself. Here I share the code and its ingredients, and would love to read comments or suggestions for potential improvements. The code is open (available from this GitHub repository) and can be used by anyone freely.
What will SteemSTEM do with such reputation indicators for now? Well, I don’t know (yet!), but it was fun to develop ;)
A good reputation for the SteemSTEM community must in my opinion include two ingredients, an authorship component and an engagement component, contributing in equal parts. Indeed, we absolutely need both authors to provide amazing contributions, and readers questioning authors and entertaining worthy discussions. I suspect that holds true for any community by the way.
AUTHORSHIP INDICATOR
The authorship metric is built from a few key principles.
Each SteemSTEM vote on a post at x% gives x reputation points at the time of the vote. The SteemSTEM curation team scours Steem to find the best STEM content. As all these blogs contributed to what SteemSTEM has become today, it makes sense they all enter any given reputation indicator.
It would be weird to assign a large reputation score to someone who contributed a lot two years ago but then left Steem. However, the reputation of that person should be somehow non negligible. After all, this person has left a trace in our memories. For this reason, I decided to introduce reputation points which vary with time. After a given time (the authorship point half-life), one point loses half its value. After twice the time, the remaining 0.5 points are 1/4 point worth, and so on. A simple exponential decay.
As SteemSTEM strives to push for quality as much as possible,e prefer someone writing one excellent post a week (thus supported very strongly) over someone writing 5 good posts a week (thus supported five times moderately). For this reason, the reputation score today (i.e. accounting for the fact that each gained point has lost value with time) is divided by the square root of the number of posts. The square root tames the effect when a very large number of posts is reached.
Finally, we may want to remove individuals from the algorithm, like the team, blacklisted people, bots, etc. Moreover, the total amount of reputation points is fixed to a given value so that each score is renormalized at the end of the day.
RESULTS FROM THE STEEMSTEM AUTHORSHIP INDICATOR
I adopted an authorship half-life of 3.5 months and excluded all team members (management and curators), bots and blacklisted authors from the run. The total number of available authorship reputation points is normalized to 1000.
The top 30 most reputed SteemSTEM authors of all time, out of 2662 authors, are (with their score):
1 abigail-dantes 6.305
2 chloroform 6.244
3 egotheist 5.918
4 scienceblocks 5.884
5 steemit-italia 5.691
6 lordneroo 5.632
7 zen-art 5.216
8 nonzerosum 5.089
9 highonthehog 4.931
10 conficker 4.902
11 nikolanikola 4.893
12 effofex 4.808
13 anaestrada12 4.804
14 tomastonyperez 4.555
15 deathbatter 4.466
16 samminator 4.456
17 hidden84 4.385
18 agmoore 4.361
19 romulexx 4.357
20 jfermin70 4.266
21 answerswithjoe 4.159
22 dysfunctional 4.093
23 elvigia 4.047
24 n4zrizulkafli 4.021
25 alexander.alexis 4.017
26 dedicatedguy 3.993
27 lupafilotaxia 3.958
28 anasav 3.909
29 scienceangel 3.856
30 irelandscape 3.809
The code has been run on Sep 24th at 10:11:25 AM.
ENGAGEMENT INDICATOR
Here, I track every single comment to any SteemSTEM-supported post and give reputation points to the comment author.
First, if the comment length is smaller than N characters, it is considered as spammy and no po. Moreover, if the comment has been posted more than W weeks after the SteemSTEM vote, no point is given. I want meaningful comments that help illustrating that supported posts are interesting during the time in which they are hot or trending (on the #steemstem tag).
If non zero, the score is given by the square root of the comment length. The square root allows once again to make a large difference between smallish and average comments, but tame down the difference once a given length is crossed. This is the only way I have found so far to deduce the score, and I am only partially satisfied with it. But at least, it provides some level of quantification of the engagement of the readers.
As with the authorship indicator, any earned engagement point loses value with time, the score today is divided by the square root of the number of comments and some individuals can be removed from the algorithm.
The final score is normalized as for the authorship case, the total number of available points being fixed to a given value (taken to be the same as engagement and authorship are considered as important).
RESULTS FROM THE STEEMSTEM ENGAGEMENT INDICATOR
I adopted an engagement half-life of 1.75 months, and excluded comments whose length is smaller than 100 characters (N=100). I fixed W to 2 weeks. I excluded all team members (management and curators), bots and blacklisted authors from the run. The total number of available engagement points is 1000.
The top 30 most engaging SteemSTEM comment authors of all time, out of 23134 comment authors, are (with their score):
1 erh.germany 2.886
2 agmoore 2.764
3 steemit-italia 2.623
4 amestyj 2.621
5 abigail-dantes 2.357
6 scienceblocks 2.175
7 fran.frey 2.149
8 insight-out 2.114
9 rudyardcatling 2.096
10 lupafilotaxia 2.079
11 dedicatedguy 2.031
12 samminator 1.979
13 tsoldovieri 1.950
14 alexander.alexis 1.921
15 cyprianj 1.853
16 herbayomi 1.847
17 tomastonyperez 1.833
18 jamalgayoni 1.756
19 steepup 1.726
20 alexdory 1.682
21 kimberlylane 1.678
22 synick 1.665
23 olamseu 1.656
24 emperorhassy 1.628
25 lucylin 1.625
26 osariemen 1.611
27 ied 1.576
28 egotheist 1.575
29 delpilar 1.526
30 chireerocks 1.487
The code has been run on Sep 24th at 10:11:25 AM.
FINAL REPUTATION INDICATOR
The final reputation is given by the average of the two above metrics. The top 25 (with the score) is given by
1 abigail-dantes 4.331
2 steemit-italia 4.157
3 scienceblocks 4.029
4 egotheist 3.746
5 chloroform 3.579
6 agmoore 3.563
7 lordneroo 3.489
8 nonzerosum 3.247
9 erh.germany 3.245
10 samminator 3.217
11 tomastonyperez 3.194
12 conficker 3.151
13 effofex 3.115
14 lupafilotaxia 3.019
15 dedicatedguy 3.012
16 alexander.alexis 2.969
17 anaestrada12 2.959
18 nikolanikola 2.911
19 cyprianj 2.795
20 tsoldovieri 2.734
21 zen-art 2.723
22 jfermin70 2.605
23 alexdory 2.598
24 highonthehog 2.576
25 amestyj 2.572
The code has been run on Sep 24th at 10:11:25 AM.
MORE ABOUT THE CODE
The code can be obtained from the following GitHub repository. It is programmed in Python 3 and requires steem-python.
I am not happy with the way the engagement indicator is computed, because I need to get the information on each post separately, which takes an enormous amount of time. For this reason, the information is saved into a file when the SteemSTEM upvote on a post is older than two weeks (as any later a comment would just bring 0 point). This requires removal of the ‘null’ author from the algorithm, which is used to trace posts without any single comment.
To run it, it is sufficient to complete the setup part of the code,
## Setup
half_life_vote = 3.5*30*24*3600. # 3.5 months - authorship point half-life
half_life_comment = 1.75*30*24*3600. # 1.75 month - engagement point half-life
comment_timelimit = 14*24*3600. # 2 weeks - the W number
comment_spam_limit= 100 # minimum number of characters for a comment to be valid (N)
comment_filename = 'comments_data.txt' # where to save the treated comments
load_backup = True # Using the file with the saved comments
normalized_rep = 1000 # Score normalization
## Exclusions
team = [ ‘null’ ]
bots = [ ]
blacklist = [ ]
and execute the program.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Hey, I didn't know you would review this. Thanks for the comments. We are now several of us working on an improved version of this. I am sure all of this will be accounted for!
Thank you for your review, @helo!
So far this week you've reviewed 7 contributions. Keep up the good work!
I am not sure what your final usage of this.
It seems heavily dependent on the curation team, not the STEM community but that might fit the use case.
A few days ago @heimindanger posted about the Dlive case and a comment he made against them. Anyone who upvoted that comment were then blacklisted for votes at the suggestion of one of the Dlive team (redjepi). This seems to be a risk of sorts here too where the personal relationships between poster and curator will effect decisions heavily in both the positive and negative, regardless of content quality. Does anyone who questions STEM or give any negative view then suffer at the hands of vindictive curators?
Even if a post itself is brilliant and attracts high engagement from the STEM community, it isn't factored in unless it gets a vote and, that size of vote is dictated by the curation team itself.
I think the curators/management should be included in the calculations for transparency purposes also as they factor into the voting and quality of STEM posts.
My concerns are things for example like:
The top 30 most reputed SteemSTEM authors of all time, out of 2662 authors
15 deathbatter 4.466
This person who I found interesting got support from their first post by the looks and all stem posts but hasn't posted at all in 2 months. They have only been registered for 4. Shouldn't reputation also come with some length of track record? Perhaps the degradation takes this into account but they are still number 15 of all time and they managed that in the first 2 of their 4 months with the last 2 being absent.
I am not a coder and this is far out of my area of expertise so perhaps it suits the purpose it is intended for.
What I do like is the degradation which should be applied to witness votes to stop inactive witnesses still holding top 50 positions. I also like that there are many people working on possible solutions to what is the useless Steem rep system. Having many people think from different perspectives means that there is a chance that something decent is developed.
Question and experimentation is good right?
If a reputation system like this takes it'll be an excellent motivator for engagement and quality posts. Awesome work!
My only worry is that it might devalue the contribution of longstanding members that post occasionally or members taking a break from SteemSTEM. Maybe if there was a limit to how much an old post's value could decay (like 1/8 or 1/16 of it's original value)? Then again, I wouldn't want to discourage new members so maybe it's not the best idea.
Yeah I had a similar thought about 'historical' authors, whose rep like lemouth said should be non-negligible; but then how can you do that when you introduce the half-life? Perhaps a different metric altogether for historical authors? Perhaps a tally should be taken every week or month, and then those tallies get tallied?
In other words, say steemSTEM has existed for 10 weeks. A historical member has been active for the first 5 weeks and then quit. A newer member has been active for the last 3 weeks. Both have contributed equally while they were active, getting the maximum score of 1 for every week of activity. So the historical contributor would get 5 out of 10 and the newer member would get 3.
I'm basically trying not to exclude the St. Anselms of steemSTEM :D I mean, the rep of people like JTM who are seldom publishing now will be effectively 0 in the long term by the current metric. He specifically is excluded by the code since he's a founder, but the future might give us non-founder examples.
And now I'm just typing away to increase my score. Nothing to see here. Move along. Just finger-dancing on the keyboard... la la la la la... :D
A long letterhead automatically affixed to all my comments now seems like a great idea...
Tagging @tking7798, @tarazkp and @justtryme90 as this may be relevant as an answer to their own comments.
This "max loss" thingie does not change much for the historical author, and actually, the good idea would be to introduce a way to back-trace long term authorship. For the engagement part, I think we all agree we should focus on the recent stuff.
So how to do that in practice? Well I don't know.
Any thoughts?
Yep, that makes sense.
Other than that, I can't add much to the technical conversation :)
If the point of the reputation score is only to decide how to vote on the latest post, then it does make sense to privilege the recent stuff. I think I was thinking it more like an overall badge of merit (basically, a replacement of the steem rep number, since you're making an independent website) and felt awkwardly about discounting the historical contribution of valuable members who for one reason or another are not as present, currently, on the platform.
For the moment, the score is just something funny. Like you said, a kind of badge of merit. This is by no means affecting any future vote as the rules for votes are independent (and clearly stated).
The decaying reputation is more to have a way to slowly remove users that have left. There is a difference between not being present and having left, which is why the half-life is large. However, this may be improved (cf. the above comment by @tarazkp).
It does not change much for the historical author now, but what about after we've been running for 5 years?
I can't think of a better normalizing function to apply in place of the sqrt, given the nature of people and high volume posting. It does a good job of squashing that while bringing up the people who post few posts. Other functions would perhaps open up the middle more, but I don't see how it would help old posters.
Just to come back to this: I am planning to implement a time dependent half-life. In practice, at the time of a vote at x%, x/i% of the value lost by a point is recovered with i being a parameter to fix. I will try that later this month.
Curious to see how this changes the rank ordering.
I have considered different half-life values before picking the above ones for the exact problematics you were mentioning. However, I have never thought about a maximum loss of value. I will implement this "maximum decay" after work, and see what we get. According to the results, we could decide whether this is a good idea or not (I think it is).
Maximum decay seems like a good idea to me, so long as the value is relatively small, maybe even less then 1/16th of its original value. This criterion is likely more important for bloggers then for commenting. As we want to encourage active engagement rather then past engagement.
I agree. It shouldn’t be so large that it discourages new users from contributing and it may not work out in the end. Either way, I’m super psyched you all liked my idea :)
I have tried 1/8 and 1/16. 1/16 has an effect on what is going on beyond rank #75, the first modification being the position #78. 1/8 start affect anything beyond #10. However, the modifications sound really wrong compared to my gut feelings. Like users not active for more than a year are there... I am thus keeping the 1/16 for the next version.
You are #56 for the authorship metric and #80 for the comments, in case you are interested. This gives a final position of #50 (#51 with the original code).
That was fast work. Interesting to see where both methods start having an effect. Thanks for the ranking info too :)
I like to try things, and in any cases, many brains are always superior to a single brain (which I why I always share my ideas).
That s a great initiative and I like the introduction of the exponential decay.
I m not sure I like the normalization though, users should be motivated to see their reputation go up, but in this case it can go down just by having more users.
To make it even more complete, you could use the UA score of the post rather than its number of votes/payout.
Thank for the comment. It makes me thinking about improving the algorithm :)
I agree this is great (and motivating) to see the rising evolution of the reputation score. However, I really want that the two factors, engagement and authorship, contribute equally. Consequently, one is forced to normalize somehow. Maybe a good solution would be to take the total sum of the two indicators of all users, divide by two and normalize each indicator to this number. I will definitely try that tonight. Note that by virtue of the exponential decay, the score can actually go down no matter what.
I do not like the UA score because it measures somehow the connection to the top witnesses, which is not appropriate for a community. Here, we have a community account (@steemstem), with a behavior driven by the rules behind the community, that I wanted to use as a seed for the metric.
Note that by virtue of the exponential decay, the score can actually go down no matter what.Yep but in this case it's a "deserved" decrease of the reputation IMO.
Maybe the number of users could also be a factor in your calculation.
I do not like the UA score because it measures somehow the connection to the top witnesses, which is not appropriate for a community.I have voiced similar concerns to @scipio since it is a very centralized way to start regarding the concentration of VESTS in the hands of the few. However, the way I understand it, it was only used to initialize the algorithm, so the importance of top witnesses should decrease over time (TBC).
But right, if you already have your own community calculation, that works!
For the engagement indicator, I instead use the length of the comment as a seed for the score (which is questionable, but I haven't found any better option so far).Among the many projects that I want to work on but can t possibly the find the time for it, I was thinking that developing a spam detector shouldn t be too hard using machine learning and SteemPlus to let users provide samples of spammy content.
The ML thing would be great. Now, I only have an antispam filter as a minimum length for a comment. This is not optimal but allow to get rid to a large amount of spammy comment.
Am still thinking about the normalization thing. @justtryme90 is right and we don't want this to be a competition. It is just a funny metric and that's it.
I ll keep the ML on my TODO list then ^^
A bit of competition is healthy IMO as long as it doesn t become too important.
Fully agreeing here :)
Well here they would want to rise in the reputation ranking. So it would almost be a competition among users to push themselves ahead in the leader board.
Gosh @lemouth, after reading the comments on this feed I feel I don't really have anything else to add. I share @tking's concern, but I have seen the solution you presented to him, and look forward to hearing the final decision on the maximum decay idea.
You guys definitely run a tight ship here, and it is wonderful to see how far the recognition from the SteemSTEM team goes towards those who are committed to the community. People respond really well to this sort of reinforcement. It is without a doubt a very efficient way to obtain desirable outcomes (quality + meaningful engagement).
:)
Unfortunately, this only affects anyone beyond rank 75. Therefore, this is not a good solution and more brainstorming is in order. You may want to check the first comments (I tried to upvote them kind of strongly to make them visible). Any extra opinion is welcome :D
And thanks for your contributions by the way, constant after all these years ;)
Oh, how interesting. Funnily, before you were suggesting to read your article in the discord chat, I had kind of a similar question and voila, you came out with this. Synchronisity ;-)
Unfortunately I am not very good in doing numbers and leave it up to more skilled people to give you feedback on how logic and sensible your approach is.
But I like this idea for motivation. Thanks for your work!
I kind of assumed that my engagement in commenting is quite high but like it to be confirmed. I would really appreciate statistics like this. This is actually also a good tool for self-observation and to check if what I estimate relates to the "true" numbers.
It would have been a shocking surprise if you were not the top comment author of this community Erika! I immensely respect (and appreciate) the depth with which you engage with the posts you comment. Not only do you leave your heartfelt contribution to the overall message; but, you never fail to bring new information and add invaluable meaning to the discussion :)
Thank you, Abi!
I am glad about your appreciation. It surprised me nevertheless a bit as I did not so much engage for quite some time in the steemstem community as I was widening my interest also to other communities.
You yourself are a high scorer. Which I would have thought so, too. :)
It eats up a lot of time, though and taking a break and rest is needed once in a while. That's why I thought @tking77798 s argument should also be considered.
After all, quality writing and commenting (in the sense of relating to the given content and invest energy in it) requires a lot of resources in both time & energy.
@tking77798 argument is considered. I just don't know how for now... Brainstorming is in order :)
Hehe! Are we brain-synchronized? I will not repeat what @abigail-dantes said, but you deserve your #1 position for engaging the SteemSTEM community members. It was actually not a surprize to get you there automatically by the algorithm. The opposite would have be :)
I am still thinking about ways to ameliorate the code, but I am pretty sure you will keep your number-1 slot ^^
LOL! For a short timespace our brains somehow synchronized magically.
Though, I guess, having me as a commenter is not always a pleasure ;-)
Looking forward to the final of your programming.
In some time... I cannot tell when but there is no deadline here, so... :)
Is fabulous all the time and work that you dedicated to the creation of this program, it is good that STEM users realize that we are monitoring absolutely everything in the community and this type of statistics is an incentive for people to try to interact with comments a little more than usual.
The growth that STEM is showing as the months pass is great :D
People have indeed no idea on the amount of scripts we have :D
Hey, @lemouth!
Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the development category on Utopian for being of significant value to the project and the open source community.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
Wow, this was not expected! Thanks guys!
Hey, cool! This is a excellent idea. Really nice to see how engagement effects the overall score. Great work, thanks!
You are welcome. I am still unsure this is the best way to monitor engagement. But so far, I haven't found anything better. Any suggestion? :)
No clue, what you've good here looks good but it is missing the quality/helpfulness/"STEMness" of the comments. Although I'm not sure that's a necessary part of it as long as the discourse is good.
This is true, but you then may need machine learning and things like that. I don't want to go that far :D
Hi @lemouth, nice post.
My question is how do you treat bought bids? People using Bidbots are actually paying for bids and is thus gaining reputation, which they did not earn. In theory I can buy myself into the top reputation spot by buying a lot of votes!! This is against the principal of good content being rewarded!!
This is not possible here, as only posts voted by @steemstem and the value of the sole @steemstem upvote conribute to the reputation score. In this way, even if they are 1324 other votes to a post, they just give a sum of 0. This ensures that reputation cannot be obtained otherwise than through:
I hope this clarifies, otherwise please come back to me :)
Thank you, it does clarify.