Today I Learned…

Bi-directional sync between Mastodon and Twitter using Github Actions

After setting up my Mastodon instance, I wanted to set up bi-directional sync between Mastodon and Twitter. I wanted to be able to post to either platform, and have it show up on the other. Andrew recommended mastodon-twitter-sync, but like the Mastodon instance itself, I didn’t want to have to operate a server tu run this on. Github Actions is great for things like this: though it bills itself as a CI/CD tool, it’s also a great way to run random bits of code on a schedule.

In the end this is pretty simple, though I ran into issues setting up the Twitter integration (more on that below), but overall it’s pretty straightforward.

The only potentially-hard part is storing state. mastodon-twitter-sync writes a state file (post_cache.json) that it uses to track which posts have already been synced. The “trick” I used here is something I learned from Simon Willison: commit the changed state file back to Github if it changes.

With that idea in place, here’s what I did:

• Ran the tool locally, first, to get authorization set up and create an initial post cache:

  docker run -it --rm -v "$(pwd)":/data klausi/mastodon-twitter-sync --skip-existing-posts

  This prompts for various credentials from Mastodon and from Twitter. [I really struggled with the Twitter part, and documented that in detail below], but I’m not sure if this was just me being dense or if it’s a broader issue.

  I wanted --skip-existing-posts because I didn’t want to sync anything written before I set this up.

• This creates a mastodon-twitter-sync.toml config file and a post_cache.json file.

  I edited the config file, setting mastodon.sync_reblogs and twitter.sync_retweets to false. This prevents syncing of retweets/boosts. My reasoning here is that people publishing content to either platform may not want it synced to the other one, and it’s not really mine to sync. The other defaults seemed fine to me.

• Created a new private github repository, and committed these two files.

  The one thing that’s gross about this approach is that mastodon-twitter-sync.toml contains Twitter and Mastodon secrets. Committing these to Github isn’t great. A potential improvement would be to put these secrets into Github’s actions secret store, and write them to the config file at runtime. But I think I trust a private repo well enough for my purposes here.¹.

• Created a github action to run these commands on as an action. Here’s my workflow file:

  name: sync
  
  # Run every 15 minutes, or when manually triggered
  on:
    schedule:
      - cron: "*/15 * * * *"
    workflow_dispatch:
  
  jobs:
    sync:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v3
  
        - uses: addnab/docker-run-action@v3
          with:
            image: klausi/mastodon-twitter-sync
            options: -v ${{ github.workspace }}:/data
            run: mastodon-twitter-sync
  
        # Commit the updated post_cache.json if it's changed
        - run: |-
            git config user.name "🐦🐘🤖"
            git config user.email "[email protected]"
            git add -A post_cache.json
            timestamp=$(date -u)
            git commit -m "sync @ $timestamp" || exit 0
            git push          
  

  This uses addnab/docker-run-action to run the mastodon-twitter-sync Docker image, and Simon’s trick to commit the updated post cache back to Github if it’s changed.

  Adding on.workflow_dispatch allows me to manually trigger the action – e.g. with gh workflow run sync.

Issues/Caveats

This is now working, and running, and posting both ways. I have noticed two issues/caveats:

1. Mastodon has a 500 character limit, but Twitter’s is 280 (and mastodon-twitter-sync further limits this to 240 because apparently there are intermittent API errors as you get close to 280). This means that long posts will be truncated on Twitter. I don’t love this, but it’s fine.
2. I don’t love the way quote-tweets are synced to Mastodon - it posts the contents of the quoted tweet, which means truncation often, I don’t love it (example). Issue #17 tracks this. I may try to cut a PR some day: I think it’s straightforward enough.

Other options

I had several people recommend moa.party to me. It’s a hosted tool so looks like it would have been a million times easier. I vaguely didn’t love giving posting permissions to a third party, but I think I’m just being paranoid. For most people this is probably a better approach.

Setting up the Twitter API app

As mentioned above, I had a hell of a time getting a working Twitter integration. The docs for mastodon-twitter-sync say that all you have to do is make a new app and paste in the keys, but that just kept not working for me – I got various authentication errors from Twitter. I could get a read-only API key issued, which would work for syncing Twitter -> Mastodon, but for the life of me I couldn’t figure out how to get a write-enabled key. I flailed around for several hours before finally finding something that worked. I hope this is just something weird about my Twitter developer account (which I’ve had forever), or that I was doing something wrong. But in case I have to do this again, here’s all the steps I did:

1. Make sure I didn’t have a config file (as if I was starting from scratch)
2. docker run -it --rm -v "$(pwd)":/data klausi/mastodon-twitter-sync --dry-run to start the auth dance. --dry-run is necessary to avoid syncing boosts/retweets before I have a chance to turn that off.
3. Do the Mastodon auth parts, then leave the CLI prompt running when I get to the Twitter auth parts.
4. Create a “Project” (not just an “App”; a standalone app never worked for me) if I don’t already have one
5. On that project page, click “Add app”. Select “Create a new app”, and “Production” on the next two screens.
6. Name the app: “Mastodon Twitter Sync”
7. Note down the API key and secret but don’t enter them in the CLI yet – these are readonly creds so far, and if I authorize against them I get a readonly authorization, and mastodon-twitter-sync doesn’t know how to elevate credentials.
8. Click next, which goes to the app details page. Under “User Authentication Settings”, click “Set Up”
9. Select “Read/Write” for permissions, and “Native” for type of app. OAuth URLs are required; I used https://social.jacobian.org/ for both which is a lie (it’s not a valid OAuth callback URL at all), but it worked. Click “Save”.
10. Now enter the API key and secret from step 7 into the CLI. mastodon-twitter-sync calls them consumer key and consumer secret.
11. Click the provided authorization link, complete the auth dance, and post the PIN back into the CLI.
12. Edit mastodon-twitter-sync.yaml, disabling retweets/bosts, as described above.
13. Finally run again, either without --dry-run to do an actual sync, or with --skip-existing-posts as described above to start syncing from this point.
14. Remember to commit the config and post cache back to Github so the action has the right credentials.

If anyone knows why this was such a pain in the ass, I’d love to hear about it.