Skip to content

Job sensor is aiming for inaccurate window of upstream #640

@arinda-arif

Description

@arinda-arif

Describe the bug
A daily job that depends on another daily job, does not have an accurate sensor aiming for the same data window. For example, given this case:

Job A

  • Window size: 24h
  • Truncate to: d
  • Offset: 24h
  • Schedule: every 22.00

Job B

  • Window size: 24h
  • Truncate to: d
  • Offset 0
  • Schedule: every 23.00
  • Have dependency to job A

Job B will have a sensor that is pointing to job A, with the same window. However, let's say on 2nd October run, job B sensor is checking on job A 1st October run, instead of 2nd. The impact is if Job A on 2nd October takes longer to finish (for example 23.30), job B will not wait for it to be finished as the 1st October run has been finished.

To Reproduce
Create 2 jobs with details as mentioned in the bug description.

Expected behavior
Sensors should be created accurately. Daily jobs depend on another daily job should wait for an equal data window.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions