Socket Mode disconnection issue with the aiohttp-based client

[matthieucan]

Hello,
I'm following up after the issue #1065 and upgrading to Slack Bolt 1.8.

Shortly after upgrading, I experienced disconnections. This happened twice in the last 3 days - with the previous version the disconnection has happened every few weeks.
What's visible from my side is a Slack application still running, but unreachable through Slack slash commands.
There are no logs related to the websocket, contrary to #1065. The bug symptoms are the same though, which makes me believe it's related.

Reproducible in:

The Slack SDK version

slack-bolt==1.8.1
slack-sdk==3.10.1
slackclient==1.3.2

Python runtime version

Python 3.8.10

OS info

(running in a Docker container based on python:3.8.10-slim-buster)

Steps to reproduce:

Run the app with socket mode for a few days.

Expected result:

The app is always reachable.

Actual result:

Slash commands or interacting with the app messages lead to the Slack error "/foo failed with the error "dispatch failed".

Thank you for your consideration!

[seratch]

Hi @matthieucan, thanks for taking the time to report this issue (again)! I'm sorry to hear that the fix we had didn't help for your case.

I will try to reproduce the situation and do more testing on my end but can I ask you to help us identify the cause of the issue by running a simple app with debug level logging? Having the following lines of code at the top of your source code enables the debug level logging.

import logging
logging.basicConfig(level=logging.DEBUG)

If you find any suspicious behaviors in the detailed logs, sharing the outputs with us would be greatly appreciated.

[matthieucan]

Hi @seratch, thank you, I'm now running it with debug logs. I can't guarantee I'll let it run until the bug is triggered, as this application is critical enough to warrant reverting the Slack Bolt upgrade. I'll let you know!

[seratch]

Thanks for your help and we're sorry about the disruption of your apps due to this issue. I have been running a simple Socket Mode app in the same Docker image for two days but I haven't managed to reproduce the situation yet. I'm thinking that some race conditions that can arise only when an app is receiving more requests.

FWIW, the following is the example app I'm using now:

main.py:

import logging
logging.basicConfig(level=logging.DEBUG)

import os
from slack_bolt.async_app import AsyncApp
app = AsyncApp(token=os.environ["SLACK_BOT_TOKEN"])

@app.command("/do-something")
async def handle_some_command(ack, body, logger):
    await ack()
    logger.info(body)

from slack_bolt.adapter.socket_mode.async_handler import AsyncSocketModeHandler

async def main():
    handler = AsyncSocketModeHandler(app, os.environ["SLACK_APP_TOKEN"])
    await handler.start_async()

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Dockerfile:

FROM python:3.8.10-slim-buster as builder
RUN apt-get update && apt-get clean
COPY requirements.txt /build/
WORKDIR /build/
RUN pip install -U pip && pip install -r requirements.txt

FROM python:3.8.10-slim-buster as app
COPY --from=builder /build/ /app/
COPY --from=builder /usr/local/lib/ /usr/local/lib/
WORKDIR /app/
COPY *.py /app/
ENTRYPOINT python main.py

requirements.text:

slack_bolt>=1.9
aiohttp>=3,<4

[matthieucan]

Hi @seratch,
Thank you for trying to reproduce along!
Not much to update on my side, I've been running the app with debug logs for the last 2 days, but the bug didn't happen.

I'm thinking that some race conditions that can arise only when an app is receiving more requests.

This looks correct. For the record I have a copy of the app running in a development cluster, that doesn't receive nearly as many requests, and it never broke like the prod one.