👨💻 Steel Orca Tech Support #1: I broke Substack (and it wasn't that hard)
👨💻 Steel Orca Tech Support #1: I broke Substack (and it wasn't that hard)
📖 How a technology company's production support operates is a key indicator of the health of their overall business. Last week suggests Substack's model is broken. It begs us to ask, "What else is?"
Welcome to the inaugural issue of Steel Orca Tech Support. In this series, we document the technical challenges we encounter in our day-to-day dealings with the internet and technology. We are not claiming expertise, only seeking knowledge and wisdom. If you’re reading this and have the answers we don’t, leave a comment below.
In my Songs of the Week #13 post, I stated the following:
I had some technical issues with Chrome after Substack made the latest update, but I didn’t get anywhere with Substack’s ticketing system or Twitter, so I switched to Edge. A bit disappointing, but the platform is free (beyond my paying for the domain, which was a one-time deal). I’m getting what I pay for, and other people were at least not reporting the problem, so maybe it’s all my fault.
That ended up being somewhat prescient, as upon posting on Friday, November 16 at 10 am MT, that post broke both the iOS app and the Android app. At the time of writing, Songs of the Week #13 still will not render on the Android app.
To direct my energy toward something positive, I thought it would benefit everyone if I documented my experience with Substack technical support. The two resources I’m aware of for end users are the Substack Status page (to monitor incidents) and the Substack Help Center, which is powered by Zendesk.
In addition, while I’m not a DevOps or IT support person by profession, I do “know enough to be dangerous,” as every middle-manager IT hack says multiple times per week. So I have also applied severity (S1-4) and priority (P1-4) as rough estimates.
Finally, these posts will be living documents, meaning, if an incident is still ongoing at the time of publishing, I will still publish, and the post will just be updated (without re-sending to everyone) based on subsequent updates that I receive or gather through additional testing (to the extent I do any). I hope you enjoy and find this helpful, and I will try to keep it positive.
Essential Context: “Substack is powered by Ghost.”
On December 12th, John O’Nolan, Founder/CEO of Ghost (one of Substack’s main competitors), posted the following Twitter thread:
I am not going to go through it and explain the whole thing. But, essentially, it looks like the Substack development team got caught red-handed, effectively copying large amounts of Ghost’s code (and not attributing it correctly) to make a big splash with the release of Bari Weiss’ new publication, The Free Press.
At the end of the thread, O’Nolan calls out Substack’s co-founder Chris Best. Chris replied in a thread of his own:
In my opinion, Chris’s response was less than convincing. This tweet from him near the end gives you the gist:
To which this random dude cheekily replies:
Again, this is essential context based on when the issues below started. I understand they may be and likely are separate. Still, from what I can tell, the Substack development team is under intense pressure to deliver new platform features as quickly as possible. This is especially true in light of the additional context that Twitter has indicated they will shut down their newsletter service Revue on January 18, 2023. If I were a Substack executive, I’d think it’s time to strike while the iron’s hot.
Current Bugs
One note on the bugs below - from what I could tell, the Substack Help Center does not generate any ticket number or request ID for the customer upon submission or when Zendesk generates the automated support email. This doesn’t seem great. The only way I got the ticket numbers below was by going into the email HTML and getting the support ID that’s auto-generated for the “Reply To” email (see below):
That’s not something anyone will do, so upon submission, you cannot track a follow-up on your help desk ticket. Send one and pray. Hopefully, your issue isn’t too serious. Not a great look.
S3P2 - “Duplicate to drafts” freezes Substack editor in Google Chrome (Current Status = Open)
Ticket# 231591 - Support Request Received Dec 18, 22:38 MT
The first thing to mention is that this was the original bug I encountered last week, but I wanted to do my due diligence for the Substack team and create and document a support ticket before making this post. So the tickets were submitted out of order, but chronologically this is how it went down.
As mentioned above, Substack has been releasing a lot of new features. One of which was “Duplicate to drafts”:
That tweet excited me as my Songs of the Week series has a standardized format I manually recreate each week. This would save me a bunch of time.
Except when I went to click on it and use the functionality, it didn’t work. Immediately my browser froze. I tried it again, thinking it was an issue with my machine’s performance, and still nothing. So I tried it in another Chrome session, and it froze immediately.
Whenever I tried to duplicate a post, it would reproduce successfully as an object on my writer’s dashboard, but it would be utterly unopenable in Chrome (even to delete). After complaining to Substack on Twitter about this (and the “Subscribe with Caption” button issue described below), I then figured out a workaround where if I opened the same duplicated post in Google Chrome’s mobile browser on my phone, I could at least get to the section where I could delete the posts. So I did that, given that I didn’t want a bunch of garbage drafts sitting on my Substack writer’s dashboard.
After doing that, Substack replied on Twitter:
At that point, I just left it as I figured, “Substack is free, don’t worry too much about it; you’ve got a workaround.”
S3P2 - “Subscribe with Caption” button freezes Substack editor in Google Chrome (Current Status = Open)
Ticket# 229777 - Support Request Received December 14, 15:45 MT
When I discovered the “Duplicate to drafts” issue above, I figured, “Well, who cares? I didn’t have this functionality before today, anyways. I’ll do what I planned and recreate the post manually.”
To drive subscriptions (as one does on Substack), I include a “Subscribe with Caption” button near the top of each newsletter. No big deal. Why don’t you try the one below to see how easy it is if you haven’t used one before?
Unfortunately, my editor session completely froze when I clicked on the “Insert subscribe button with caption” on the Editor toolbar. I’ve just tried it in Chrome and am still having the same issue. However, everything seems to work just fine in Microsoft Edge.
S2P2 - Post Does Not Render on Android Substack App (Current Status = Open)
Ticket# 230613 - Support Request Received December 16 10:48 MT
… and this is where the proverbial sh*t hit the fan, as it were. Because of the above technical issues, I had to work on Friday’s post until around midnight on Thursday, then went to bed and worked on it another couple of hours until just before posting time. Having switched to Edge, everything seemed fine. I sent myself a few draft posts, and it seemed all good. Ok, great.
I completed my final reviews of the post at around 9:45 am MT, scheduled it to send out at 10 am MT (9 am PT, 12 pm ET) to keep it consistent, and then went and did some other work.
I get the email to both my Android and iPhone, and everything seems fine. The email looks excellent, and everything seems to work.
At around 10:15 am MT, Clara Beth (my publishing partner in charge of photography) then messaged me saying the iPhone post was rendering weird and that I should check it out. So here’s what I got:
I then tried the same post on the Android app, and it just caused the Substack app to quit altogether. Unfortunately, Android doesn’t surface the error details, but the assumption is that it’s a similar issue.
No one is going to paint me as some IT guru, but here’s what all that’s telling us - Substack is not returning valid JSON for a post that was able to be successfully created and published via their in-browser editor, which is one of their primary value drivers.
At this point, I started blasting Substack on Twitter because Friday is the only day I publish, and this was going to kill my view count (spoiler alert: it did). So I DM’d Substack and posted a few times but didn’t see any reply (and they were continuing to retweet advertisements for random new accounts during this time). So not having any luck, I decided to go straight to the top:
Now Hamish is a good guy and was a total gentleman about the whole thing. When I posted the actual JSON error above, he reacted quickly:
I don’t really like Twitter, to be honest, but as you can see above, this was the quickest way to get someone of some import at Substack actually to look into the issue. And my line above about “Maybe you guys imported too much of the Ghost code without checking it?” is not a nice thing to say (especially for a Canadian), but it’s lines like these that get eyeballs on Twitter, and thus, reactions from the bigwigs.
So Hamish talked to his development team, and they seemed to indicate an issue with emojis rendering on their app. That suggests that the problem was just introduced during the last major update (which happened around December 10, from what I can tell), as including emojis in all of my posts is standard, and none of them have had issues except this last post. Songs of the Week #13 isn’t even my most emoji-heavy post, so something substantially changed in the code so that emojis cause it to not render. The weird thing is my old posts with emojis still work fine.
I would strip out the emojis as Hamish recommended, but I’m not sure that’s truly the issue, and that’s fundamentally asking me to make a design change to my newsletter to accommodate undocumented tech limitations on his newsletter platform. That doesn’t seem right to me. Furthermore, if I change the post, I will have no idea when an update has been pushed that resolves the issue, which I can then update everyone here on. This bug is a secret portal to the inner workings of Substack’s Android application development team.
I continued to test the post on both iPhone and Android throughout the day because, even though this is a free “hobby” newsletter, I’m trying to do something here, and this stressed me out. Finally, about two hours later, the error was resolved for the iPhone, and that started to work, so half the problem was resolved. At the time of writing, however, this post still doesn’t render on Android. I’m assuming that’s because Swift allows quicker updates than Android, but I’m not an expert. If you know, leave a comment below.
One last update from this weekend - Substack Support replied to this ticket at around 5 pm on Saturday, December 17, with the following, which seems to indicate there has been a surge in support requests recently (emphasis mine). It looks like they chose their words carefully with the word contact:
Hi there,
Substack Support here. Thanks so much for reaching out, and apologies for the delay in response! We've had a higher volume of contact than usual and have been working to get to your message as quickly as possible.
I've reported this issue to our engineers so they can look into this further. Hopefully, they'll have an update soon.
We appreciate your patience and apologize for the inconvenience.Best,
Substack Support
Implications
Misleading Incident Reporting
At the time of writing, the Substack Status page, which, again, is seemingly the only customer-facing page providing such information, has the following detailed as “Past Incidents,” which also apparently includes current or ongoing incidents:
The fact that no incident has been listed when I have confirmation from the Substack co-founder Hamish McKenzie himself acknowledging there’s an issue (as well as the potential cause) on December 16, as well as a similar confirmation from Substack Support on December 17, indicates a fundamental disconnect in the management of incident reporting at Substack, or what information they are choosing to provide to their customer base at scale. I’ll let you be the one to decide which one it seems to be (if not both).
Editor’s Note - Before publishing, an incident for December 19 related to the editor was logged, so we’re moving in the right direction -
I checked, however, and this fix didn’t address any of my issues. I would also call out that it’s not clear from the incidents what the problem is, so this reporting is useless for the end user (especially if you’re relying on Substack for income).
Questionable Code Reviews
One of the first things people will tell you when you ask them about writing on Substack is the ease of editing and publishing. I am writing this on Substack’s editor right now. I love it and would prefer to stick with this moving forward.
The issue is that if they are breaking their editor and their posting platform while simultaneously copying code from direct competitors without proper licensing or attribution, the development team either isn’t that good or is under tremendous pressure to deliver. Who is in charge that would allow that to happen? Who is approving the pull requests? What is the review process?
In particular, in my opinion, Chris Best and Substack need to reply specifically to the tweet below. I looked today, and they have not (I assume for legal reasons, but I am not a lawyer.)
Substack’s Ability to Scale
As noted above, because Twitter announced they were shutting down Revue in early 2023, this represents a significant opportunity for Substack to become the market leader in email newsletter platforms. They are leaning into this from a marketing standpoint, but the above indicates that they are stretching their engineering and architecture teams. That’s understandable, but that shouldn’t mean introducing platform-breaking bugs and questionable code.
Final Thoughts
I like Substack. At this point, I’ve paid for a custom domain and have all of my posts (all 14 of them!) on here, hopefully generating traffic. I don’t want to leave.
However, this past week, as well as some other technical challenges that I’ll detail in further Steel Orca Tech Support posts, indicate that while Substack might be the most popular right now, it may not be the best run from a development perspective. Here’s hoping they can turn that around sooner than later.
Even though the focus at Substack is on high-quality writing, podcasting, and video publishing, it’s easy to think that Substack isn’t fundamentally a tech company. But that’s precisely what it is, and if it doesn’t start acting like it, it will lose its market share as quickly as it gains it, and people will start looking at Ghost more intensely.
definitely keeping these bugs and glitches in mind going forward... ive had no trouble viewing your posts recently, nor with my own substack publishing, but ill be keeping a careful watch out for anything similar
A very informative piece. I know nada about tech, so I'm a little lost here—still, a great read and interesting. You shed light on issues I did not know about. The illegal use of Ghost and Substack’s lack of transparency makes me question its motives. Notwithstanding Substack’s deliberate moves to draw in and keep big names on the platform, I love it here. There’s been much talk lately of Substack concentrating on the writers who have large followings and releasing features that make the site more competitive, more social-media-esque. I wonder what the new year will bring.