Song Restarted!

Support
1

Hey Leo or anyone. I had a MAJOR fail happen at an arena show on Sept. 1st. In the middle of a song, that same song restarted. I didn’t touch a thing. Both computers restarted. This of course sent timecode to all departments and the stage went black. I immediately stopped it, and I caught it back up to where the band was pretty quickly, but if this happens again I could very well lose my job. Yikes. Could you give me some insight? We have another show tonight. I’ve talked the log package from my A computer through with Claude and it seems to think that my B computer Ablenet wasn’t connected fully, and so when it did finally connect it sent a bunch of messages through. I can confirm that on my B computer it has been saying “connecting to A” in Ableset host area. It was still responding to commands so I assumed it was okay. It happened right around 9PM on Sept. 1st.

EDIT: I had attached the full log package but a kind fellow user reminded me these contain sensitive info so I will message it to support when required. Thanks!

  • OS and Version: macOS Sequoia 15.7.9
  • Version of AbleSet: 3.1.4
  • Version of Ableton Live: 12.4.3

Setup

  • Redundant Ableton Live playback, two MacBook Pros linked directly over Thunderbolt Ethernet (en14 on both), self-assigned 169.254.x addresses
  • AbleSet 3.1.4 Pro on both, AbleNet enabled, preferredNetwork blank, no custom addresses (Bonjour discovery)
  • Jump mode: quantized
  • Show control: iConnectivity mioXL (port “mioXL HST 2”, AbleNet forwarding on) with an Oaktone Mini (USB) for play / stop / prev / next, plus a Canvas page on an iPad
  • Wi-Fi off on both machines

Times converted for UTC. I was in NYC on Eastern time.

What’s in A compputer’s log (times UTC, ableset-3.1.4-2026-09-02T00-06-23Z.log)

  • 00:06:23 — AbleSet starts on A. A’s socket to B connects at 00:06:25.
  • 00:08:50 — B’s AbleSet restarts. A reconnects to B at 00:09:28.
  • 00:09 – 01:00 — A sends ~30 commands to B over AbleNet (soundcheck jumps, play, stop). In this whole window there are no “Got AbleNet message” entries from B and no “Client connected” from B’s address in A’s log.
  • 01:00:06 — Play pressed on the Oaktone (MIDI note 36), song starts. Sent to B as UUID 3ySxG8ZtvYyo2jR8joMu1B.
  • 01:02:06.111 — “Client connected” from 169.254.50.3 (B). First inbound connection from B in this session.
  • 01:02:06.375 – .455 — A logs 16 “Got AbleNet message” entries from B, then a second copy of 8 of them at .457 – .460. The UUIDs match commands A had sent to B earlier in the evening. Examples:
    • pku9P4rVQnBgUz1LBZphfV jumpToSongById 2846 — A sent this at 00:12:55
    • oxgBYY5yBbQhoYwWF3ekBj play — A sent this at 00:16:21
    • hbKmJVhoi5fQeyRqNvPmXC stop, gUjVN63rGNU3xnjLsBuziC play, 1LW6XQrWgrqXkViSUcpAS1 stop
    • grkK5itBifV3DExhzt5XNV, powyaPCSgssTRQwqBEa1Py, jA2f7ChXFXSXVs94re2kqh jumpToSongById
    • iBFZjfk4ZE3c7Ze8pUoMrK, 3X1gUKawBLS5XXyJCK7kj3, 3ySxG8ZtvYyo2jR8joMu1B play
  • 01:02:06.459 — Playback state changed: isPlaying false
  • 01:02:06.483 — Playback state changed: isPlaying true
  • 01:02:08 — My stop on the Oaktone (note 38). The jump logged two seconds later shows lastSongTime 3.3 beats past the song start.

In the same 80 ms window I don’t see any incoming MIDI, any OSC from the iPad, or anything from the other OSC sources on the rig. The only inbound traffic is the AbleNet burst from B.

Other things I noticed, which may or may not be related

  • A’s self-assigned IP on en14 was different in every log this week (169.254.130.128, 169.254.141.227, none at startup on the afternoon of Sept 1, 169.254.130.104 that night). B was 169.254.50.3 throughout.
  • Normal operation all night: every command A sent came back from B as “Got AbleNet message” within about 3 ms and was not re-executed. The burst at 01:02:06 was the exception.

What I don’t have yet

I have not been able to pull my B computer’s log bundle. The rig is packed and I haven’t had access to that machine since the show. I’ll attach it as soon as I can. Without it I can’t see what B was doing during the hour before it connected, or what triggered the connection at 01:02:06.

Questions

  1. Is the burst at 01:02:06 the cause of the restart, or could it be a symptom of something else?
  2. Does AbleNet buffer outbound messages while a peer socket is disconnected and send them on connect? If so, is there an age limit?
  3. Is the UUID dedup time-limited? The immediate echoes were ignored all night; these were not.
  4. Is there a known reason B’s connection to A would take roughly an hour? Would preferredNetwork or custom AbleNet addresses be the recommended setup for a redundant rig like this?

What I’m planning, pending your read

Static IPs on the Thunderbolt link on both machines and custom AbleNet addresses, plus checking B’s AbleNet panel before every show rather than only A’s. If there’s a better approach I’d rather do that.

2

I’m really sorry you had to deal with that in the middle of an arena show. That sounds incredibly stressful, especially with the impact on timecode, lighting and the whole production. I hope you get clear answers soon and your next shows go smoothly. :folded_hands:

Just a heads-up from a fellow AbleSet user: I’d recommend sending full log packages privately to support, as they can contain sensitive computer and network information. For the forum, I’d stick to relevant, redacted excerpts. This is just my personal recommendation.

3

Thank you for the kind words and also for the heads up! I didn’t think about sensitive info like that. I’m going to edit that. Appreciate it!

4

I forgot the most important information:

AbleSet Support Inbox
for your logs

5

Just got my B computer’s log package and this seems of note and the problem. 9:02pm is the exact time this happened during the show.

What B was doing for that hour (Sept 1, local time):

  • 8:09:25 PM — B’s AbleSet starts and sees PROD2A (A computer) via Bonjour. But Bonjour resolves PROD2A.local to 169.254.147.147. That was A’s address from 2:40 PM that afternoon (A’s afternoon log shows en14 picking it up then). By showtime A had re-assigned to 169.254.130.104. A’s own Bonjour record even says preferred_ip=169.254.130.104, but B went with the hostname lookup, which was stale.
  • 8:09:26 — B opens a socket to http://PROD2A.local, logs “Connecting took longer than 2 seconds,” and never re-resolves.
  • 8:09:28 — A connects inbound to B from 169.254.130.104. B logs “Connection suggested: 169.254.130.104” and then “Suggested connection already exists.” B had the correct address handed to it and declined to use it because it already had a socket to PROD2A.local, the one that was never going to connect.
  • 9:02:06 PM — B’s socket finally connects (the stale record aged out and re-resolved), and it flushes 53 minutes of buffered commands to A. That’s the restart.
6

Hey @pb_123, thank you for your very detailed report! I’m so sorry this happened to you and can understand that this is worrying.

Your theory was right, the socket used for AbleNet was configured to automatically re-transmit dropped commands when another host re-connects, which caused the flood of messages to your B machine.

Using a static IP address for both computers should fix this issue, but I also just released AbleSet 3.1.5 which disables this re-transmission feature.

The update also changes how the UUID deduplication works. In previous versions, UUIDs were deduplicated for a maximum time span of 2 seconds which was usually more than enough for making sure each command would only be executed once but which didn’t cover the situation you experienced.

The deduplication now keeps a list of the last 512 used UUIDs independently of when they were first received.

That’s an interesting question. My guess is that mDNS which AbleSet uses to discover other hosts on the network didn’t catch the IP address change. These mDNS records have a TTL of 75 minutes which roughly aligns with the time it took to reconnect.

I hope this helps. Please let me know if you have any further questions or feedback!

7

Leo thank you! I’ve gone ahead and setup the static IP’s earlier today and
everything connected quickly and seems to be working fine, although the
issue has only happened this once in our whole 2 months of rehearsals, so
it is hard to replicate.

I have 2 hours before the next show, would you suggest and say it’s safe to
update to 3.1.5? Or do you think I am safeguarded enough for tonight with
the static ip’s?

8

Hey @pb_123, the update is safe but changing to static IPs will solve this problem already, so you don’t have to update right before the show.

9

Wonderful. Thank you so much again and to this community.