tl;dr:
I have detected some malicious code that was injected into my Bluesky “MP follows” bot, so out of an overwhelming abundance of caution, if you used the bot I strongly recommend changing your Bluesky password.
The malicious code only impacted the server the Bluesky bot was on. Two technical experts I have consulted do not believe it is a targeted attack – and is more likely a generic tool designed to use my server to send out spam and so on. They both believe the risk of any Bluesky data having been taken is very, very low.
For the avoidance of doubt: My Substack is completely separate from this server and the technical aspects are not managed by me. So this incident is unrelated to my newsletter, and any data I hold about you on my Substack is completely unaffected, and is secure.
If you did not use the Bluesky bot then there is nothing to do or worry about.
But if you did, I strongly recommend changing your Bluesky password just in case.
The bot has now (obviously) been taken offline and the server it was running on has been shut down.
I am deeply sorry.
A longer explanation
What happened?
On the 22nd November, I launched a little bot/app/widget that I’d built. Essentially, you enter your Bluesky login credentials, and it reveals which MPs are following you on the new platform.
Yesterday, I noticed some malicious code had been injected across PHP files on the virtual private server (VPS) I was using to host it.
Unfortunately, I do not know when the code injection happened, but having consulted two independent technical experts their assessment is that it appears to be a relatively common attack.1
Essentially, it appears the malicious code is not directly related to my Bluesky bot – it is instead designed to take control of the server to commandeer it to, for example, send spam emails as part of a botnet.
The way it appears to have been injected was via a vulnerability with the blogging platform Wordpress.
Wordpress is a self-hosted content-management platform that is used by millions of independently hosted websites. The VPS I was using was literally a server I’ve had running for maybe 15-20 years, and it contained files from three or four long dead Wordpress installations, and the remnants of other coding projects over the years.
In other words, it was a similar situation to how your phone or computer can become vulnerable if you ignore the requests to install software updates for so long. Though in this case, I wasn’t ignoring any warnings. The mistake I made was in not realising that the PHP files used by these dead projects could still be sniffed out and exploited by bad actors (likely) using automated systems to find vulnerable servers.
So the malicious code could have been sitting on the server for literally years without me noticing – it was only when it prevented the Bluesky bot from working that I noticed that it had injected itself into every PHP file on the server.
Is my data at risk?
The technical experts I have consulted with, who have seen the malicious code, believe that it is highly unlikely any data was compromised.
The reason I am sending this message out, and urging you to change your Bluesky password is, like I say above, out of an overwhelming abundance of caution. And because I’m a bit paranoid.
Unfortunately, my VPS provider did not keep detailed server logs of when changes to files were made or when pages were accessed.
But I believe that the number of people even conceivably affected is likely to be small. When the bot first launched on the 22nd November, for the first several days it was working fine. I am confident the bot’s code was not compromised at that point.
It is only when the bot stopped working (which may have happened as recently as only a few days ago) that I know the malicious code was on the page. So if you entered your credentials and nothing happened when you hit “login”, that is a sign the malicious code could have been present.
When building the app, per best practice, I did not store more data than I needed to. When you logged into the app, no data from your Bluesky account was retained by my server (except very briefly in a session cookie, so the app could function). So there is no database of Bluesky login credentials that any malicious code could have downloaded and taken things from.
The only exception to this is if you had ticked the box asking me to add you to my Substack mailing list (virtually nobody did this): But even then, the only data retained was your (public) Bluesky user ID and username, and your email address.
This is a good thing, as it minimised the data I was storing. But this does unfortunately mean that I do not know how many people used the bot while it was active.
I do know that only 145 people clicked the link to the bot from my Substack post announcing it – and it didn’t get that much pick-up on Bluesky itself, so I would estimate the total number of users in the low hundreds.
However, given traffic to new posts is highest on the the first days and it quickly trails off after, my actual estimate is the number of people potentially affected by the malicious code is very, very low – perhaps numbering in the low two-digits.
And even then, there is no evidence that the malicious code had any interest in user-submitted Bluesky data.
As I say above, it appears that the (highly obfuscated) code was designed to take control of the server to send spam and so on, and in the view of the experts, it does not appear to have been a targeted attack.
So I think it is very likely that even if the bot was used while the malicious code was present, that nothing actually happened to anyone’s data.
Essentially, I’d characterise the current situation like getting home and realising that you had left the door unlocked. Everything is almost certainly fine, but you can’t say with absolute certainty whether or not anyone entered your house while you were out.
What action do I need to take?
I will say it again. Just to be on the safe side, if you used the bot I recommend changing your Bluesky password.
I am hoping that most users of the app took advantage of Bluesky’s “App Passwords” feature. This is where instead of using your main Bluesky password to login to third party apps, you can create a temporary secondary password. I flagged this up on the login screen and said it was “strongly recommended” that this method be used – but I had no way of forcing people to do this, so they could have conceivably used their real password.
If you used an app password, then you can just delete the app password in your Bluesky settings. If you used your real password – you should change it.2
And once again, for the avoidance of doubt: This is nothing to do with my Substack or (for paying subscribers) Stripe. Those are all hosted elsewhere and managed by other companies who know what they are doing. There is no risk to any data on these platforms and everything remains secure.
What steps have been taken since?
When I discovered the issue I immediately took action to disable the bot, and then the server.
My domain now points away from the VPS, and I have closed my account with the hosting provider and asked it to delete all of the files within.
And as you can see by reading this, I am now notifying you about it. Though I don’t have full details on who used the bot (see above), this has been sent out to the same mailing list where I announced the bot, and I will be linking to this post on Bluesky too.
Frankly, given that the experts view the problem as annoying but very low risk to any Bluesky data, I probably could have chosen not to post any notification, but I wanted to follow best practice about how to deal with incidents like this.
What are the lessons from this?
Though I fully own this fuck-up, I think many technically-minded people reading may be thinking “there but for the grace of God go I”, given how many of us no doubt have our own personal servers that we use for coding projects and messing around with.
The reason I was using a VPS is because this is just how things were done back when I first learned to code two decades ago - it was a classic “LAMP”3 server where I’ve hosted blogs and other widgets and toys I’ve built over the years.
But aside from the fact that I’m never, ever, going to risk running anything like this again, if there’s a more universal takeaway from this nightmare, it is to underscore the importance of keeping software up-to-date. What fucked me up here was ancient Wordpress code I’d forgotten about, which had remained unpatched.
Again, I want to emphasise how deeply sorry I am about this. I’m hopeful that no one has actually been affected by it. But I wanted to be transparent and let you know, to err on the side of caution.
Since I’ve found countless forum posts and the like with people describing falling victim to the same sorts of attacks.
I am not blaming Bluesky for this but the risk is exacerbated because it has not yet fully implemented Oauth login (though it is reportedly working on it). Oauth is the authentication system that’s used by the likes of Twitter and Facebook, where the app kicks you back to the platform, where you confirm you’re happy to share your data. That method does not involve sending passwords – instead it uses secure temporary tokens. Unfortunately, Bluesky has not yet rolled this out yet, so instead most third party Bluesky integrations (like, for example, the Graysky iPad app) have been using the same method I did.
Linux, Apache, MySQL, PHP – the four technologies that were foundational to how every bedroom coder learned to code twenty years ago.
thanks for heads up, v good of you