Remote workers

[circleconsult-mk]

Hi

I'm pretty new to background tasks on Flask, but I really need it in my current application. I like to have stuff running on any platform, and the worker doesn't seem to be supported on Windows unless we use WSL or something similar.

I have tried to get this to work with a remote worker running in Docker, my Redis stuff is started with the following composer file.

version: '3.8'

services:

  rq-dashboard:
    build:
      context: /tools/rq
      dockerfile: Dockerfile
      args:
        - IMAGE_TAG=dev
    image: rq-docker:dev
    command: rq-dashboard -u redis://rq-server:6379
    ports:
      - "9181:9181"
    restart: unless-stopped

  rq-worker:
    image: rq-docker:dev # This is the same image as rq-dashboard.
    command: rq worker -u redis://rq-server:6379 default high normal low
    deploy:
      replicas: 3
    restart: unless-stopped

  rq-server:
    image: redis:alpine
    ports:
      - "6379:6379"
    restart: unless-stopped

Here I also included the dashboard to see how things go, I've started Flask-RQ2 in my flask app with Factory Pattern, and the jobs are easily added to the queue, but does the Docker Workers need application context?

I currently just have a simple function in ../application/tasks.py

import time
from application import rq

@rq.job
def example(seconds):
    print('Starting task')
    for i in range(seconds):
        print(i)
        time.sleep(1)
    print('Task completed')

and execute it with

job = example.queue(23)

The job is queued and catch by one of the workers with the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/utils.py", line 107, in import_attribute
    return __builtins__[name]
KeyError: 'application.tasks.example

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/rq/worker.py", line 1428, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1278, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 1315, in _execute
    result = self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/rq/job.py", line 425, in func
    return import_attribute(self.func_name)
  File "/usr/local/lib/python3.10/site-packages/rq/utils.py", line 109, in import_attribute
    raise ValueError('Invalid attribute name: %s' % name)
ValueError: Invalid attribute name: application.tasks.example