Hi all,
I was thinking about a new user mode (+C) to help us in our endless fight against spamming. This user mode will only allow messages/notices/invites from users who share one or more channels with you with a few exceptions: 1) It'll not work on users who are in no channels. 2) Opers (+o) will be able to bypass it. 3) With the combination of umode +R, identified users (+r) will be able to bypass it.
It'll need to be enabled by default to be effective IMO (possibly together with umode +R so users will be able to get messages from either people who are identified or share the same channel with them).
Thoughts?
-Kobi.
(Kobi_S @ DALnet)
Personally, I think if we're going to add a new user mode, I'd prefer a mode that hides what channels you're in unless it's a common channel. Like a personal +s. The majority of the spam I see comes from in channel sources, but as an op in #help, I tend to see someone I kick from #help for being disruptive suddenly being disruptive in every channel I'm in. I think +R tends to deal with the vast majority of out of channel spam.
FDSA
On Fri, Apr 24, 2009 at 9:23 PM, Kobi Shmueli kobi@dal.net wrote:
Hi all,
I was thinking about a new user mode (+C) to help us in our endless fight against spamming. This user mode will only allow messages/notices/invites from users who share one or more channels with you with a few exceptions:
- It'll not work on users who are in no channels.
- Opers (+o) will be able to bypass it.
- With the combination of umode +R, identified users (+r) will be able to
bypass it.
It'll need to be enabled by default to be effective IMO (possibly together with umode +R so users will be able to get messages from either people who are identified or share the same channel with them).
Thoughts?
-Kobi.
(Kobi_S @ DALnet) _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
I actually really like both of these ideas (+C, and FDSA's idea of a "personal" +s). I have nothing useful to contribute at this time other than stating my support. =)
Brandon / lorddracula CSop and AA of serenity.* servers Exploits, Routing, Oper Training, and Webteam
Vin King wrote:
Personally, I think if we're going to add a new user mode, I'd prefer a mode that hides what channels you're in unless it's a common channel. Like a personal +s. The majority of the spam I see comes from in channel sources, but as an op in #help, I tend to see someone I kick from #help for being disruptive suddenly being disruptive in every channel I'm in. I think +R tends to deal with the vast majority of out of channel spam.
FDSA
On Fri, Apr 24, 2009 at 9:23 PM, Kobi Shmueli <kobi@dal.net mailto:kobi@dal.net> wrote:
Hi all, I was thinking about a new user mode (+C) to help us in our endless fight against spamming. This user mode will only allow messages/notices/invites from users who share one or more channels with you with a few exceptions: 1) It'll not work on users who are in no channels. 2) Opers (+o) will be able to bypass it. 3) With the combination of umode +R, identified users (+r) will be able to bypass it. It'll need to be enabled by default to be effective IMO (possibly together with umode +R so users will be able to get messages from either people who are identified or share the same channel with them). Thoughts? -Kobi. (Kobi_S @ DALnet) _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net <mailto:DALnet-src@lists.dal.net> https://lists.dal.net/mailman/listinfo/dalnet-src
DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
Kobi Shmueli wrote:
Hi all,
I was thinking about a new user mode (+C) to help us in our endless fight against spamming. This user mode will only allow messages/notices/invites from users who share one or more channels with you with a few exceptions:
- It'll not work on users who are in no channels.
- Opers (+o) will be able to bypass it.
- With the combination of umode +R, identified users (+r) will be able
to bypass it.
It'll need to be enabled by default to be effective IMO (possibly together with umode +R so users will be able to get messages from either people who are identified or share the same channel with them).
Thoughts?
-Kobi.
I really like this idea, this would take away the ability of spammers to use relays and let channel operators remove spammers without having to rely on opers to track down the relays.
However, upon giving this more thought... would it not be easier to automatically set +R on connection?
Kevin Buley wrote:
I really like this idea, this would take away the ability of spammers to use relays and let channel operators remove spammers without having to rely on opers to track down the relays.
However, upon giving this more thought... would it not be easier to automatically set +R on connection?
It would be easier but there are a lot of users who don't have a registered nick and probably don't want one either. They just want to chat and we shouldn't take that away from them.
-Kobi_S.
I think it's a great idea. Lots of people want to us to improve how we are tackling spam, and if this can help reduce the amount people see its worth doing. It's true, lots of people don't like to register their nicks (often due to laziness I've found) but if they can get into a channel the person they want to message is in, then they won't have issues.
Naturally, as already mentioned, we will need to put help and info all over the place, but I'm sure that won't be a problem - a simple link to a page on the website in many languages will be one step we can take.
Just my thoughts - RedPepper
On Mon, Apr 27, 2009 at 9:55 PM, Kobi Shmueli kobi@dal.net wrote:
Kevin Buley wrote:
I really like this idea, this would take away the ability of spammers to use relays and let channel operators remove spammers without having to rely on opers to track down the relays.
However, upon giving this more thought... would it not be easier to automatically set +R on connection?
It would be easier but there are a lot of users who don't have a registered nick and probably don't want one either. They just want to chat and we shouldn't take that away from them.
-Kobi_S. _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
Hey all, The idea is excellent. IMHO, how about we can have it implemented first and run it on a user discretion basis. Enforcing it upon successful connection can be discussed anytime once it is done.
Regards,
White`Shark
On Tue, Apr 28, 2009 at 10:57 PM, RedPepper RedPepper@dal.net wrote:
I think it's a great idea. Lots of people want to us to improve how we are tackling spam, and if this can help reduce the amount people see its worth doing. It's true, lots of people don't like to register their nicks (often due to laziness I've found) but if they can get into a channel the person they want to message is in, then they won't have issues.
Naturally, as already mentioned, we will need to put help and info all over the place, but I'm sure that won't be a problem - a simple link to a page on the website in many languages will be one step we can take.
Just my thoughts
- RedPepper
On Mon, Apr 27, 2009 at 9:55 PM, Kobi Shmueli kobi@dal.net wrote:
Kevin Buley wrote:
I really like this idea, this would take away the ability of spammers to use relays and let channel operators remove spammers without having to rely on opers to track down the relays.
However, upon giving this more thought... would it not be easier to automatically set +R on connection?
It would be easier but there are a lot of users who don't have a
registered
nick and probably don't want one either. They just want to chat and we shouldn't take that away from them.
-Kobi_S. _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
It would be easier but there are a lot of users who don't have a registered nick and probably don't want one either. They just want to chat and we shouldn't take that away from them. -Kobi_S.
Yes, and they have friends to who they may want to talk with. Blocking those messages by default may hurt a lot of people. I suggest making a +S mode for spamfilter. Which will block messages based on a list of flexible criteria, and based on the user's "history", i.e. reputation of their ip address.
The problem is this reputation can't merely be stored on one server, all servers need immediate access to it; it needs to apply to all the IP address' IRC sessions, and needs to persist, even if they reconnect (otherwise spammers will /reconnect to reset their reputation).
And exempt registered, identified users get excluded from being blocked based on 'reputation'. A database of IP addresses should be held on the servers to reflect on the "reputation" of a non-registered user's IP.
I suggest a RDBM be used here. When a user connects, IRCD makes an asynchronous DB request for the IP address's "reputation" data; which is then cached somewhere in their user structure and re-synced at random (but long) intervals.
As long as their IRC session is still open there should be a 'session' entry in the database, and a separate "persistent" entry. When an IRC session is closed, and at prescribed intervals, the "persistent" entry is updated and session stats are cleared. The user's stats at any moment are the sum of all 'session' stats and the persistent stats.
DB Performance would need careful consideration. Possible elements of reputation score:
x How old is the reputation info? (old info is meaningless) x How many total private /MSG commands has the IP issued in the past 72 hours? x How many did they receive from users of high reputation (and identified users), and what's the SENT:RECEIVED ratio? (People that get a lot of messages are conversing, not spamming) x Other commands to track usage on: private /NOTICE, /JOIN, /INVITE x After removal of color codes and non-printables: x How many /MSGs did the user send that contain "http://" in them? x How many /MSGs did the user send that contain "//" in them? x How many /MSGs did the user send that contain "/eval" in them? x How many /MSGs did the user send that contain "/join" in them? x How many /MSGs did the user send that contain "/server" in them? x How many /MSGs did the user send that were blocked?
I suggest adding a /REPORTSPAM <nick> command which increments a "spam reported" reputation counter and effects their reputation score.
But if the target has not sent a private /MSG/NOTICE/INVITE in the past 2 minutes, or recipient had not received a private /MSG/NOTICE/INVITE from anyone since their last /REPORTSPAM, the reporter's reputation is penalized with a "bad report" count (negating all further spam reports from the ip for 24 hours).
-- -J
So what happens then if you're afk for a few minutes, come back, have received spam in the meantime, and then /reportspam? Also, since all traffic to channels is privmsg, would that open anyone chatting in a channel up for reports? What if a user happens to just post a lot of http:// links, like stuff from 4chan? I don't particularly see this being an effective methodology, and technically much more prone to being designed wrong than +C.
On Thu, Apr 30, 2009 at 9:03 PM, James Hess mysidia@gmail.com wrote:
It would be easier but there are a lot of users who don't have a
registered
nick and probably don't want one either. They just want to chat and we shouldn't take that away from them. -Kobi_S.
Yes, and they have friends to who they may want to talk with. Blocking those messages by default may hurt a lot of people. I suggest making a +S mode for spamfilter. Which will block messages based on a list of flexible criteria, and based on the user's "history", i.e. reputation of their ip address.
The problem is this reputation can't merely be stored on one server, all servers need immediate access to it; it needs to apply to all the IP address' IRC sessions, and needs to persist, even if they reconnect (otherwise spammers will /reconnect to reset their reputation).
And exempt registered, identified users get excluded from being blocked based on 'reputation'. A database of IP addresses should be held on the servers to reflect on the "reputation" of a non-registered user's IP.
I suggest a RDBM be used here. When a user connects, IRCD makes an asynchronous DB request for the IP address's "reputation" data; which is then cached somewhere in their user structure and re-synced at random (but long) intervals.
As long as their IRC session is still open there should be a 'session' entry in the database, and a separate "persistent" entry. When an IRC session is closed, and at prescribed intervals, the "persistent" entry is updated and session stats are cleared. The user's stats at any moment are the sum of all 'session' stats and the persistent stats.
DB Performance would need careful consideration. Possible elements of reputation score:
x How old is the reputation info? (old info is meaningless) x How many total private /MSG commands has the IP issued in the past 72 hours? x How many did they receive from users of high reputation (and identified users), and what's the SENT:RECEIVED ratio? (People that get a lot of messages are conversing, not spamming) x Other commands to track usage on: private /NOTICE, /JOIN, /INVITE x After removal of color codes and non-printables: x How many /MSGs did the user send that contain "http://" in them? x How many /MSGs did the user send that contain "//" in them? x How many /MSGs did the user send that contain "/eval" in them? x How many /MSGs did the user send that contain "/join" in them? x How many /MSGs did the user send that contain "/server" in them? x How many /MSGs did the user send that were blocked?
I suggest adding a /REPORTSPAM <nick> command which increments a "spam reported" reputation counter and effects their reputation score.
But if the target has not sent a private /MSG/NOTICE/INVITE in the past 2 minutes, or recipient had not received a private /MSG/NOTICE/INVITE from anyone since their last /REPORTSPAM, the reporter's reputation is penalized with a "bad report" count (negating all further spam reports from the ip for 24 hours).
-- -J _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
Not only that, I think using RDBM will have a drastical impact on the performance. Not only it'll slow down a connecting client but it will also need more resources than now. And we should not forget about drones/botnets as well. I too agree with Vin King and would not think that /reportspam would be a good idea. We can't even follow a general rule which is to join #operhelp for IRCOp assistance but join their private channels, how can we make proper use of /reportspam :P. Nevertheless saying, that's my personal opinion.
Regards,
White`Shark.
On Fri, May 1, 2009 at 11:22 AM, Vin King vin.king@gmail.com wrote:
So what happens then if you're afk for a few minutes, come back, have received spam in the meantime, and then /reportspam? Also, since all traffic to channels is privmsg, would that open anyone chatting in a channel up for reports? What if a user happens to just post a lot of http:// links, like stuff from 4chan? I don't particularly see this being an effective methodology, and technically much more prone to being designed wrong than +C.
On Thu, Apr 30, 2009 at 9:03 PM, James Hess mysidia@gmail.com wrote:
It would be easier but there are a lot of users who don't have a
registered
nick and probably don't want one either. They just want to chat and we shouldn't take that away from them. -Kobi_S.
Yes, and they have friends to who they may want to talk with. Blocking those messages by default may hurt a lot of people. I suggest making a +S mode for spamfilter. Which will block messages based on a list of flexible criteria, and based on the user's "history", i.e. reputation of their ip address.
The problem is this reputation can't merely be stored on one server, all servers need immediate access to it; it needs to apply to all the IP address' IRC sessions, and needs to persist, even if they reconnect (otherwise spammers will /reconnect to reset their reputation).
And exempt registered, identified users get excluded from being blocked based on 'reputation'. A database of IP addresses should be held on the servers to reflect on the "reputation" of a non-registered user's IP.
I suggest a RDBM be used here. When a user connects, IRCD makes an asynchronous DB request for the IP address's "reputation" data; which is then cached somewhere in their user structure and re-synced at random (but long) intervals.
As long as their IRC session is still open there should be a 'session' entry in the database, and a separate "persistent" entry. When an IRC session is closed, and at prescribed intervals, the "persistent" entry is updated and session stats are cleared. The user's stats at any moment are the sum of all 'session' stats and the persistent stats.
DB Performance would need careful consideration. Possible elements of reputation score:
x How old is the reputation info? (old info is meaningless) x How many total private /MSG commands has the IP issued in the past 72 hours? x How many did they receive from users of high reputation (and identified users), and what's the SENT:RECEIVED ratio? (People that get a lot of messages are conversing, not spamming) x Other commands to track usage on: private /NOTICE, /JOIN, /INVITE x After removal of color codes and non-printables: x How many /MSGs did the user send that contain "http://" in them? x How many /MSGs did the user send that contain "//" in them? x How many /MSGs did the user send that contain "/eval" in them? x How many /MSGs did the user send that contain "/join" in them? x How many /MSGs did the user send that contain "/server" in them? x How many /MSGs did the user send that were blocked?
I suggest adding a /REPORTSPAM <nick> command which increments a "spam reported" reputation counter and effects their reputation score.
But if the target has not sent a private /MSG/NOTICE/INVITE in the past 2 minutes, or recipient had not received a private /MSG/NOTICE/INVITE from anyone since their last /REPORTSPAM, the reporter's reputation is penalized with a "bad report" count (negating all further spam reports from the ip for 24 hours).
-- -J _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
On Thu, Apr 30, 2009 at 8:48 PM, Omar Sayeed Khan omar.sayeed@gmail.com wrote:
Not only that, I think using RDBM will have a drastical impact on the
Er.. so what is a drastic impact I wonder? If it takes 300 milliseconds longer to connect to IRC, would that be bad?
My thinking is most legitimately users generally stay connected to IRC, and rarely reconnect randomly (except when they're done for a while).
Usually the Reverse DNS query takes pretty long, >30 seconds, and the answer to an asynchronous DB query would almost certainly arrive back before long the DNS timeout or normal answer...
In fact, that would allow the queries to be queued.. If load was too high to immediately answer all queries; the ones that already got a DNS answer could be prioritized, so there would be less noticeable impact on connection time.
The users whose queries were delayed are the same as the ones who already delayed connecting (due to their ISP's slow reverse DNS servers)
The DNS query _is_ a database query, and it generally has to be made to a server that isn't even local.
The primary server resource utilized for a database like this is disk space and disk IOPs to read any un-cached data that is required.
I expect the DB would have at most a few hundred thousand entries.. even the most inefficient query strategy should be able to answer thousands of questions in less than a second, on 5-year-old server harware...
It's not like IRC servers see millions of unique ips in a limited time period or have 1000+ connections per second per server.
-- -J
On Thu, Apr 30, 2009 at 8:22 PM, Vin King vin.king@gmail.com wrote:
stuff from 4chan? I don't particularly see this being an effective methodology, and technically much more prone to being designed wrong than +C.
If designed correctly, I expect a lot less likely to stop legitimate traffic than +C. Just because someone didn't register their nick, doesn't mean they should necessarily be unable to talk to any of their friends...
Reputation based on IP address is a pretty common technique whose validity has been proven, especially in the operation of mail servers. Granted it has some limitations... it doesn't save you from the first spam message, and some spammers may have a few dozen IPs available.
There are certain similarities between a /MSG on IRC, and an E-mail message.
Behaviors of e-mail users can be statistically classified, so can behaviors of IRC users...
Multiple criteria allows you to increase your certainty. "Just pasting a URL" in a channel alone doesn't brand you a spammer, but other characteristics of your behavior could, or the fact you ONLY or MOSTLY pasted URLs.
So what happens then if you're afk for a few minutes, come back, have received spam in the meantime, and then /reportspam? Also, since all traffic
Then your report is in error, because it's not timely, thus your reports are useless, someone different may be using the nickname in a few minutes -- the assumption here is no one report is significant.
It's 10+ users reporting you as spam that starts to impact that aspect of your score.
to channels is privmsg, would that open anyone chatting in a channel up for reports? What if a user happens to just post a lot of http:// links, like
I would expect channel would count for something, but it's a different kind of traffic.
I suggested some sort of spam-filter being built into the next version of Bahamut to Kobi Shmueli during our conversation last week. A problem that arose was that it would be moderating IRC, which is what DALnet is not aiming on doing. However, I was trying to think of a new solution and came up with a list of defined phrases in a centralized location, so that all IRC servers on the network can access the same file, and a spam-filter module on the IRC daemon, if a user says a phrase that matches a phrase listed in the file then it sends a global warning to all operators, or operators on the server that the user is connected to warning them of a possible spammer. This wouldn't be considered moderating IRC, it is just giving IRC operators on the network an aid to work on combatting the problem of spam and advertising on the network.
My original suggestion was when a user says a phrase that matches a phrase listed in the file then the user gets auto-killed off the network. The problem with this would be quite a quantity of innocent users would be getting auto-killed from the network, which at the end of the day is only creating more work than is currently needed.
However, James I like you're suggestion, it is similar (yet a bit more complex) than mine. I definitely think that a solution like either mine or yours has to be implemented on DALnet sometime soon, spam and advertising is very high on the network, which is not unusual for such a large network like DALnet, but at the same time, something should be done to combat it, for the users.
Regards.
On 1 May 2009, at 02:03, James Hess wrote:
It would be easier but there are a lot of users who don't have a registered nick and probably don't want one either. They just want to chat and we shouldn't take that away from them. -Kobi_S.
Yes, and they have friends to who they may want to talk with. Blocking those messages by default may hurt a lot of people. I suggest making a +S mode for spamfilter. Which will block messages based on a list of flexible criteria, and based on the user's "history", i.e. reputation of their ip address.
The problem is this reputation can't merely be stored on one server, all servers need immediate access to it; it needs to apply to all the IP address' IRC sessions, and needs to persist, even if they reconnect (otherwise spammers will /reconnect to reset their reputation).
And exempt registered, identified users get excluded from being blocked based on 'reputation'. A database of IP addresses should be held on the servers to reflect on the "reputation" of a non-registered user's IP.
I suggest a RDBM be used here. When a user connects, IRCD makes an asynchronous DB request for the IP address's "reputation" data; which is then cached somewhere in their user structure and re-synced at random (but long) intervals.
As long as their IRC session is still open there should be a 'session' entry in the database, and a separate "persistent" entry. When an IRC session is closed, and at prescribed intervals, the "persistent" entry is updated and session stats are cleared. The user's stats at any moment are the sum of all 'session' stats and the persistent stats.
DB Performance would need careful consideration. Possible elements of reputation score:
x How old is the reputation info? (old info is meaningless) x How many total private /MSG commands has the IP issued in the past 72 hours? x How many did they receive from users of high reputation (and identified users), and what's the SENT:RECEIVED ratio? (People that get a lot of messages are conversing, not spamming) x Other commands to track usage on: private /NOTICE, /JOIN, /INVITE x After removal of color codes and non-printables: x How many /MSGs did the user send that contain "http://" in them? x How many /MSGs did the user send that contain "//" in them? x How many /MSGs did the user send that contain "/eval" in them? x How many /MSGs did the user send that contain "/join" in them? x How many /MSGs did the user send that contain "/server" in them? x How many /MSGs did the user send that were blocked?
I suggest adding a /REPORTSPAM <nick> command which increments a "spam reported" reputation counter and effects their reputation score.
But if the target has not sent a private /MSG/NOTICE/INVITE in the past 2 minutes, or recipient had not received a private /MSG/NOTICE/INVITE from anyone since their last /REPORTSPAM, the reporter's reputation is penalized with a "bad report" count (negating all further spam reports from the ip for 24 hours).
-- -J _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
On 25 Apr 2009, at 02:23, Kobi Shmueli wrote:
Hi all,
I was thinking about a new user mode (+C) to help us in our endless fight against spamming. This user mode will only allow messages/ notices/invites from users who share one or more channels with you with a few exceptions:
- It'll not work on users who are in no channels.
- Opers (+o) will be able to bypass it.
- With the combination of umode +R, identified users (+r) will be
able to bypass it.
It'll need to be enabled by default to be effective IMO (possibly together with umode +R so users will be able to get messages from either people who are identified or share the same channel with them).
Thoughts?
-Kobi.
(Kobi_S @ DALnet) _______________________________________________ DALnet-src mailing list DALnet-src@lists.dal.net https://lists.dal.net/mailman/listinfo/dalnet-src
That is a good suggestion Kobi_S, I know that this is already enforced on another IRC daemon and it works reasonably well and spam is near-to- none on that IRC network, it is enabled by default, which indeed as you proposed proves to be very effective. Something that you might want to consider when discussing a new usermode that will do as you propose, is the amount of users that are used to the current usermodes, and current default usermodes on connect. New users might get confused by the idea if it is enabled by default. Explanations of the new default usermode would have to be posted somewhere, and all users would have to be notified about it (especially as all users will have it on-connect).
On the network that I talked about first off, their help channel is just full of questions from new-to-IRC users asking the same question, why they aren't receiving private messages from users. Although, with the exceptions that you have proposed, I can't see any problems arising like this, just ensure that exceptions to the usermode are chosen correctly, and efficiently.
Generally, you might want to consider adding other features to the IRC daemon to prevent advertising, rather than adding something to hide it. Would this be something you would consider, if someone would propose an idea?
--- This e-mail and its attachments are intended for the above named persons only and may be confidential. Information contained should not be, may not be, disclosed to any third party without prior written consent from the above sender. If you have been the recipient of this e-mail unintentionally you should delete this e-mail, and/or any attached files to this e-mail. Please note that this e-mail has been created in the knowledge that the Internet is not a 100% secure communications medium. We advise that you understand and observe this lack of security when e-mailing us. Although steps have been taken to ensure that this e-mail and its attachments are virus free, we advise that the recipient should take steps to ensure they are virus free.
-
James Hess -
Kevin Buley -
Kobi Shmueli -
lorddracula -
Omar Sayeed Khan -
RedPepper -
Steffan Wood -
Vin King