What’s new in PyLink – 2018-07-18

After one complete year and over 900 commits, PyLink 2.0 has finally reached its release candidate stage. 2.0-rc1 was tagged today and is now available in Git master (PyPI will be updated for the final release).

Upgrading

For those who have installed using Git, upgrading is as simple as switching to the master branch (git checkout master), upgrading your copy of the sources (git pull), and re-running setup.py.

Future 2.0.x updates will be released solely in the master branch, so those currently on devel will want to switch back to master as work on the next release has not yet begun.

Effectively, this post is a call for testing – while 1.x configuration files will work, you will likely want to adapt it for 2.0 (e.g. for renamed option) or start from scratch with a new configuration.

Summary of 2.0 Changes

From the 2.0-rc1 release notes:

New features

  • Added support for ngIRCd, ChatIRCd, and beware-ircd (via protocol modules ngircdts6, and p10 respectively)
  • Add support for extbans on UnrealIRCd, Charybdis (and derivatives), InspIRCd, and Nefarious.
  • U-lined services servers can now be configured for use with Relay:
    • CLAIM restrictions are relaxed for service bots, which may now join with ops and set simple modes. This prevents mode floods when features such as DEFCON are enabled, and when a channel is accidentally registered on a network not on the CLAIM list.
    • DEFCON modes set by services are ignored by Relay instead of bounced, and do not forward onto other networks unless the setting network is also in the channel’s CLAIM list.
    • To keep the spirit of CLAIM, opped services not in a channel’s CLAIM list are still not allowed to kick remote users, set prefix modes (e.g. op) on others, or modify list modes such as bans.
  • New Antispam plugin, with the ability to kill / kick / block mass-highlight spam and configured “spam” strings.
  • Added TLS certificate verification, which is enabled by default on Clientbot networksissue#592
    • This adds the options ssl_validate_hostname and ssl_accept_invalid_certs options which have defaults as follows:
      • Server type ssl_validate_hostname ssl_accept_invalid_certs
        Full links (S2S) false (implied by ssl_accept_invalid_certs: true) true
        Clientbot networks (C2S) true false
      • ssl_validate_hostname determines whether a network’s TLS certificate will be checked for a matching hostname.
      • ssl_accept_invalid_certs disables certificate checking entirely when enabled, and also turns off ssl_validate_hostname.
      • The existing TLS certificate fingerprint options are unchanged and can be turned on and off regardless of these new options.
  • New Relay features:
    • LINKACL now supports whitelisting networks in addition to the original blacklist implementation (see help LINKACL). issue#394
    • Relay IP sharing now uses a pool-based configuration scheme (relay::ip_share_pools), deprecating the relay::show_ips and relay_no_ips options.
      • IPs and real hosts are shared bidirectionally between all networks in an ipshare pool, and masked as 0.0.0.0when sending to a network not in a pool and when receiving those networks’ users.
    • KILL handling received a major rework (issue#520:
      • Instead of always bouncing, kills to a relay client can now be forwarded between networks in a killshare pool (relay::kill_share_pools).
      • If the sender and target’s networks are not in a killshare pool, the kill is forwarded as a kick to all shared channels that the sender has CLAIM access on (e.g. when they are the home network, whitelisted in CLAIM, and/or an op).
    • relay_clientbot now supports setting clientbot styles by network. issue#455
    • The defaults for CLAIM (on or off) and LINKACL (whitelist or blacklist mode) can now be pre-configured for new channels. issue#581
    • New relay::allow_free_oper_links option allows disabling oper access to CREATE/LINK/DELINK/DESTROY/CLAIM by default
    • relay_clientbot: add support for showing prefix modes in relay text, via a new $mode_prefix expansion. issue#540
  • Clientbot is now more featureful:
    • Added support for IRCv3 caps account-notifyaccount-tagaway-notifychghostextended-join, and userhost-in-names
    • Clientbot now supports expansions such as $nick in autoperform.
    • Configurable alternate / fallback nicks are now supported: look for the pylink_altnicks option in the example config.
    • Added support for WHOX.
    • Clientbot can now optionally sync ban lists when it joins a channel, allowing Relay modesync to unset bans properly. See the fetch_ban_lists option in the example config.
    • Failed attempts to join channels are now logged to warning. issue#533
  • New commands for the opercmds plugin, including:
    • chghostchgident, and chgname, for IRCds that don’t expose them as commands.
    • massbanmasskillmassbanre, and masskillre, which allow setting mass bans/kills/G/KLINEs on users matching a [email protected] mask, exttarget, or regular expression. The hope is that these tools can help opers actively fight botnets as they are connected, similar to atheme’s clearchan and Anope’s chankill commands.
    • checkban and checkbanre, which return the users matching a target
  • Messages sent by most commands are now transparently word-wrapped to prevent cutoff. issue#153
  • PyLink accounts are now implicitly matched: i.e. user1 is now equivalent to $pylinkacc:user1.
  • PyLink now responds to remote /STATS requests (/stats cu, and o) if the stats plugin is loaded.
  • The PyLink service client no longer needs to be in channels to log to them.

Feature changes

  • The ratbox protocol module has been merged into ts6, with a new ircd: ratbox option introduced to declare Ratbox as the target IRCd. issue#543
  • The raw command has been split into a new plugin (plugins/raw.py) with two permissions: raw.raw for Clientbot networks, and raw.raw.unsupported_network for other protocols. Using raw commands outside Clientbot is not supported. issue#565
  • Some options were deprecated and renamed:
    • The p10_ircd option for P10 servers is now named ircd, though the old option will still be read from.
    • The use_elemental_modes setting on ts6 networks has been deprecated and replaced with an ircd option targeting charybdis, elemental-ircd, or chatircd. Supported values for ircd include charybdiselemental, and chatircd.
  • The fml command in the games plugin was removed.

Bug fixes

  • clientbot: fix errors when connecting to networks with mixed-case server names (e.g. AfterNET)
  • Fix irc.parse_modes() incorrectly mangling modes changes like +b-b *!*@test.host *!*@test.host into +b *!*@test.hostissue#573
  • clientbot: fixed sending duplicate JOIN hooks and AWAY status updates. issue#551

Internal improvements

  • Reading from sockets now uses a select-based backend instead of one thread per network.
  • Major optimizations to to user tracking that lets PyLink handle Relay networks of 500+ users.
  • Service bot handling was completely redone to minimize desyncs when mixing Relay and services. issue#265
    • This is done via a new UserMapping class in pylinkirc.classes, which stores User objects by UID and provides a bynick attribute mapping case-normalized nicks to lists of UIDs.
    • classes.User.nick is now a property, where the setter implicitly updates the bynick index with a pre-computed case-normalized version of the nick (also stored to User.lower_nick)

What’s new in PyLink – May 26, 2018

A busy first year of university has finally concluded… Now, here is a look at what’s been happening in PyLink development over the last half year. To keep this post at a manageable length, I will only be focusing on the new stuff released this month. (The development branch meanwhile has gotten more updates since than just that; you can find the interesting bits of 2.0-alpha1 and 2.0-alpha2 in their release notes.)

2 releases in May 2018

Two new releases were pushed this month, 1.3.0 in the stable series and 2.0-alpha3 in the development branch.

PyLink 1.3.0

The 1.3 series focuses on backporting some commonly requested features from the 2.0 (devel) branch.

So far, this has included a backport of the 2.0 launcher, bringing in daemonization support, the ability to ignore stale PID files, and shutdown/restart/rehash support via the command line.

As well, 1.3 adds the ability to restrict login blocks to opered users, specific networks, and specific user hosts. Service bot fields and relay server suffix can now be set by network, and relay gained support for a few more modes such as InspIRCd blockhighlight +V and exemptchanops +X.

These changes are documented more precisely in the 1.3.0 release notes.

PyLink 2.0-alpha3

2.0-alpha3 was not a massive release feature-wise, though some highlights include it being the first 2.0 snapshot to include the daemonization support now ported to in 1.3, as well as a new (alpha-quality) antispam plugin specifically targeting mass-highlight spam. Other minor details include Clientbot now supporting expansions such as $nick in autoperform, and better STATUSMSG handling in relay.

This release instead focused on reworking several parts of PyLink’s core, including sockets, service bots, mode handling, and user tracking. I will list some of the major bits here:

Porting PyLink to select

In 2.0-alpha3, PyLink’s socket handler was ported to use select instead of one listener thread per network. (More specifically, it now uses the selectors module in Python 3.4+, which is a higher-level frontend to epoll/poll/select/etc.)

Because this port is still somewhat new, I have noticed some instability when it comes to handling disconnects. In particular, there are some sporadic issues with autoreconnect not firing that still need to be investigated. That said, I cannot say if the 2.0 stack is more or less stable than the 1.3 one since I have been running 2.0 snapshots on my production instance for quite some time. (As always, testers are welcome!)

Reworking channel handling in service bot handling (API break)

Prior to this release, service bot channel handling and relay were a complete mess. Referring back to issue#265, PyLink 1.x had sore points where service bots wouldn’t rejoin relay leaf channels on kill and lingered around when a channel was delinked. Later 2.0 snapshots had the opposite issue where service bots left the main (hub) channel when any leaf network was delinked.

After some thought, my solution to these problems was to make the entire notion of “plugins making service bots join channels” a plugin-specific mechanism. In the end, 2.0-alpha3 introduced several new functions: ServiceBot.add_persistent_channel(), ServiceBot.remove_persistent_channel(), ServiceBot.get_persistent_channels() which use namespaces to track which channels different plugins want a service bot to be in.

ServiceBot.join() was repurposed to explicitly join service bots to channels (without remembering them as persistent), while ServiceBot.part() was added to trigger a part request, where a service bot would only actually leave a channel after no plugin marked it as persistent. Kill and kick handling in services_support were similarly reworked to use the new get_persistent_channels(), which also adds in config-defined service bot channels to any set by plugins.

Another somewhat experimental change is that service bots now try to be dynamic: that is, they will leave persistent channels once they become empty and rejoin them whenever someone else does. No considerations have been made yet to restore channel timestamps and reverse channel takeovers etc., since channel protection is somewhat out of our scope as an extended service.

Fixing edge cases in mode parsing (issue#573)

In PyLink versions before 2.0-alpha3, attempts to prevent modesetting loops  caused an issue where mode changes that both set and unset a particular mode dropped the unsetting portion when parsing. That is, if you tried to set +b-b *!*@test.host *!*@test.host or something similar, plugins like relay would see it as +b *!*@test.host only and leave you with a pretty confusing desync.

This problem was fixed in 2.0-alpha3 by reworking the mode parser to apply mode changes incrementally (onto a temporary mode state) as each single mode change (or internally, “mode pair”†) was read. This allows the check that prevents removing nonexistent modes to always have an up to date reference when it decides whether to keep or drop a specific mode change.

Work still needs to be done to potentially backport this to 1.x, though doing so is somewhat difficult because the changes are invasive (and because core function names in 1.x and 2.x no longer match up).

† Mode pairs are how mode strings in PyLink (and other projects like Supybot/Limnoria) are internally represented. The format is simple, with individual mode such as +o james split into a 2-item tuple: ("+o", "james††"). If a mode takes no argument, the second item is None (null).

†† Actually, PyLink would store the argument for a status mode change as a UID identifier†††, which stays consistent through nick changes and the like.

††† When this isn’t possible, numbered “Pseudo-UIDs††††” are used instead.

†††† Just kidding, there are enough footnotes here.

Seriously optimized user tracking

This alpha snapshot brought about significant optimizations to how users are tracked. Previously, each network’s users index only contained only a single mapping from user IDs to nicks, while going in reverse (nick→UID) required a reverse lookup through the entire user list every single time. This was a giant misdesign on my part because it simply did not scale for networks going up to a few hundred users.

Fortunately, I realized a quick and elegant fix by overwriting the setter for the each user object’s ‘nick’ attribute to write to a nick→UID mapping for us. Eventually, that became irc.users.bynick, which tracks UIDs by normalized (lowercased) nicks and is completely abstracted away from PyLink protocol modules.

What’s new in PyLink: August 15, 2017

PyLink 1.2.0 released!

PyLink 1.2.0 was released today to PyPI and Git master. You can find the full changelog on GitHub: https://github.com/GLolol/PyLink/blob/master/RELNOTES.md#pylink-120

Updated builds for Debian and Ubuntu will be made available soon.

New stuff in 2.0-dev

In the meantime, PyLink 2.0-dev received a bunch of new features. These include:

  • Limiting login blocks by network, hostnames, and IRCop status: see the new require_oper“, “hosts“, and “networks” options in the example config.
  • Support for ngIRCd and ChatIRCd, following another large refactor of the protocols stack to reduce code duplication (compare!).
  • New commands for the opercmds plugin, including:
    • chghost, chgident, and chgname, for IRCds that don’t expose them as commands. Thanks to Mitchell Cooper for the patch!
    • massban, masskill, massbanre, and masskillre – these commands allow setting kickbans, kills, or glines on users matching a certain PyLink mask ([email protected] mask or exttarget) or regular expression (regexp). The hope is that these tools can help opers actively fight botnets as they are connected, similar to atheme’s clearchan and Anope’s chankill features.
    • checkbanre – the regexp-based equivalent to checkban
  • Extban support for quiet (UnrealIRCd ~q:, InspIRCd m:) and nick change bans (UnrealIRCd ~n:, InspIRCd n:)  in relay.
  • Better support for (pre-defined) services servers in relay:
    • CLAIM restrictions are relaxed for service bots, which may now join with ops and set simple modes. This prevents mode floods when features such as DEFCON are enabled, and when a channel is accidentally registered on a network not on the CLAIM list.
    • DEFCON modes set by services are ignored by Relay instead of bounced, and do not forward onto other networks unless the setting network is also in the channel’s CLAIM list.
    • To keep the spirit of CLAIM alive, opped services not in a channel’s CLAIM list are still not allowed to kick remote users, set prefix modes (e.g. op) on others, or set list modes such as bans.
  • GLINE/KLINE support for most supported IRCds; work is ongoing to polish this off.
  • Per-network configuration of relay server suffixes; patch by Mitchell Cooper.
  • Proper separation between aliases and regular commands, with commands marked as aliases now hidden from the list command.
  • /STATS support via the stats plugin (/stats c, u, and o are supported so far)
  • Implicit syntax for PyLink accounts in host matching: i.e. “user1” is now equivalent to “$pylinkacc:user1
  • Optional display of the PyLink connection time in WHOIS, following Anope’s behaviour. This info can be turned off using the pylink:whois_show_startup_time option.
  • Support for “Network Administrator” and “Network Service” as oper types on IRCds using user modes to denote them (e.g. UnrealIRCd, TS6).
  • Fixed compatibility with hybrid trunk after commit ircd-hybrid/[email protected]981c61e (EX and IE are no longer sent in the capability list)
  • More specific permissions for the “remote” command, which now allows assigning them by target network, service, and command.

Some deprecations and backwards incompatible changes were also introduced:

  • The fml command in the games plugin was removed.
  • Signal handling on Unix was updated to use SIGUSR1 for rehash and SIGHUP for shutdown – this changes PyLink to be more in line with foreground programs, which generally close with the owning terminal.
  • PID file checking is now enabled by default: for users upgrading from PyLink < 1.1-dev, you may have to manually remove PyLink’s PID files before starting the service. Checks for stale PID files will be added at a later date, but before the eventual 2.0.0 release.
  • Some options were deprecated and renamed:
    • The “p10_ircd” option for p10 servers is now named “ircd”, though the old option will still be read from.
    • The “use_elemental_modes” setting on ts6 networks has been deprecated and replaced with an “ircd” option targeting charybdis, elemental-ircd, or chatircd. Supported values here include “charybdis“, “elemental“, and “chatircd“.

Nightly builds sunset for Debian 8/Jessie

PyLink nightly builds for Debian Jessie (which is now the oldstable release) will be discontinued on 2017-10-31. Those wishing to receive nightly builds past this date should upgrade to Stretch and use the stretch-nightlies dist instead.

Nightly builds for Ubuntu, which are provided via Launchpad, are not affected at this time.

What’s new in PyLink: July 2, 2017

What’s happening

First and foremost, cleanup work is well underway for the next release of PyLink (2.0): see issues #475, #454, #476 as specific examples. New features as mentioned in the previous post (restart command, per-network configs for relay) will likely follow suit in the devel branch as cleanup finishes up.

What to expect next

Two major things will likely arrive in the next couple of days:

  1. PyLink 1.2: the 1.2.x series will be branched off and frozen early, with all remaining new features rescheduled for 2.0 or 2.1. My rationale for this is that given the amount of invasive cleanup being done, it is much easier to write new code than to write it first and then port it across an API break. With this said, the 1.2 release will likely be the last feature release in the 1.x line.
    • PyLink 1.2.x will stay in two branches: beta (newly created!) and devel during its (probably short) beta cycle, before being merged into master in time for its RC (release candidate).
  2. PyLink 2.0-dev: The next development release of PyLink will jump straight to 2.0, as the large amount of cleanup work staged so far have made keeping API compatibility more work than worthwhile. The good news is that existing config files will still be compatible, as I will hold off on removing deprecated config bits like the bot: block until 3.x.
    • After the 1.2 release, the work staged so far for 2.0 will merge into devel and undergo active development. This means that the devel branch will become unstable and breakage prone! Users depending on 1.2 features are encouraged to move over to the beta branch, to minimize potential disruption to their networks.
    • The final list of new features for 2.0 is not set yet, but you can find the to-do list so far here.

What’s new in PyLink: June 8, 2017

This month, PyLink 1.2 continues to progress, with numerous bug fixes added and cleanup work ongoing since 1.2-alpha1:

  • Performance improvements for outgoing message queuing: the queue engine now blocks when no messages are in the queue instead of needlessly polling it multiple times a minute.
  • Added a new $and exttarget, which allows chaining multiple ban masks together in the form $and:(mask1+mask2+mask+...). The new Exttargets guide doc shows some interesting examples on how to use this.
    • Also, negating hostmasks (e.g. !*!*@localhost) is allowed now; previously this was limited to exttargets only (e.g. !$oper was OK)
  • PyLink now supports configurable per-server encodings via an aptly-named encoding option. Previously, PyLink hardcoded UTF-8 decode/encode for all text, which is now only a default.
  • Two bug fixes regarding service bot protection, which have also been backported to 1.1 Git (i.e. the master branch):
    • Fix rejoin on KILL not working at all (this was broken for months but left unnoticed, oops!)
    • Fix rejoin on KICK not working on P10/IRCu when join modes is activated (#465)
  • Various bug fixes aimed at speeding up shutdown and preventing sporadic startup freezes with the Relay plugin.

What to expect next

High up on the list of goals for PyLink 1.2 is cleaning up PyLink’s Irc/Protocol stack, which powers the core of the services framework. Specific tasks include standardizing the coding style (camel case->snake case, consistently marking private methods with _, etc.) and breaking up the Irc class into protocol-agnostic state tracker(s). The latter, when implemented, will allow future expansion to non-IRC protocols which don’t use a simple socket for communication.

Other priorities include making more parts of the Relay plugin configurable per network (e.g. clientbot styles, server suffix), and adding (likely to PyLink-Contrib) an in-place restart command. The final list of new features for 1.2 is not yet set in stone, and will likely change as new tasks are added and others get bumped between releases.

Errata

A few reports have mentioned sporadic SSL disconnections with OpenSSL 1.0.2: https://github.com/GLolol/PyLink/issues/463. Unfortunately, I have not been able to reproduce the issue, and so cannot offer a fix. OpenSSL 1.0.2 ships with Ubuntu 16.04 LTS, so the impact will likely enlarge as that adoption grows. (It is also unknown at this point which versions of PyLink this bug affects; my assumption is all of them!)

Downloads

PyLink 1.2-alpha1+ is available in the devel branch of PyLink’s source tree; you can switch to it by running git checkout devel and re-running setup.py. It is worth noting that 1.2 is in its pre-release stages, so there will probably be bugs!
Debian and Ubuntu users can also install nightly builds of the devel branch through their respective APT repositories.

TL;DR

Jjjjjohn GIF - Find & Share on GIPHY