Jump to content
Sign in to follow this  
chain

Report on the January 2020 extended outage

Recommended Posts

Last night we experienced approximately 12 hours of downtime between around 18:00 and 06:40 UTC, caused by a prolonged period of internet routing issues which our ISP has attributed to a failed line card in one of their routers. This was our longest period of downtime in many years and we’re very sorry for the disruption it caused.

Running a large service which interfaces with the venerable IRC protocol poses a different set of challenges to most modern web services: Firstly, we have to manage a large number of outbound IRC connections while ensuring as few disconnections as possible. Secondly, IRC networks expect our users to connect from a consistent set of IP addresses, and lastly, IRCCloud is subject to a high volume of distributed denial of service (DDoS) attacks.

These constraints mean that our outbound connection servers, which actually make your outbound IRC connections, have been hosted for years by a specialist DDoS-resistant hosting service provided by a major ISP. This is a costly part of our infrastructure, and it wouldn’t be economical for us to completely duplicate these servers elsewhere to mitigate against rare situations like the one last night. Switching to another ISP - even if we could find one to provide the required servers at short notice - would involve a long process of getting new IP addresses whitelisted by IRC networks.

Our current architecture also restricts us to running our outbound connection servers in relatively close proximity to the rest of our infrastructure (which is hosted on Amazon Web Services). Over the last few months we’ve been working on a significant update of our backend software to remove this restriction - in fact, we started rolling this update out yesterday.

These improvements will make it easier for us to investigate other approaches for our outbound connection servers in future, and we’ll certainly be discussing network redundancy with our ISP and future providers.

If you’re an IRCCloud subscriber, we’re happy to issue you a month’s refund in compensation for this downtime - drop us an email at team@irccloud.com.

Tuesday January 22 2019 • posted by james

Bouncer: connect with other clients

Today we’re launching one of our most requested features. Paid subscribers can now use 3rd party IRC clients to connect to the IRCCloud service, just as you would with a traditional bouncer.

Connect with another client menu item

Open the menu for one of your IRC or Slack connections and choose the “Connect with another client…” option for details on how to connect.

For IRC connections, you’ll be prompted to generate a unique server password.

Backlog replay

Note: backlog replay isn’t currently available for Slack connections

Bouncer passwords are shown to you in the following format:

bnc:xxxxxxxx…

If you’d like the bouncer to replay missed messages whenever you reconnect with your client, you’ll need to change this format to include a clientid of your choosing.

This is used to identify and track the messages your client has seen to make sure we only replay undelivered messages.

The clientid can be anything, but can’t include spaces. Just make sure to use a different id for each client you use.

Once you’ve chosen a clientid, rewrite your password in the following format:

bnc@clientid:xxxxxxxx…

For example, if your generated password was bnc:abcxyz and you chose laptop as a clientid, you’d connect with the following server password:

bnc@laptop:abcxyz

Security

A bouncer password grants full access to the associated network connection, so make sure to keep it safe.

You can revoke or regenerate a bouncer password at any time, in case you no longer need it or it becomes compromised. This will also disconnect any client currently using that password.

Backlog timestamps

The latest versions of most 3rd party clients support the server-time IRCv3 feature, which the bouncer will use to provide the correct timestamp for backlog replay.

However, some clients may need a little coaxing

https://blog.irccloud.com


Share this post


Link to post
Share on other sites
 

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

Copywrite © 2020 ChainScriptz

×
  • Create New...