Add an overwrite boolean when modifying repo content

[daviddavis]

Add a overwrite boolean parameter to endpoints which modify repository content. When set to false, the update checks whether any content being added would overwrite existing content (based on repo_key_fields) and returns a 409 Conflict response with details instead of silently replacing it.

fixes #7550

[gerrod3]

I just added the same feature (in a slightly different manner) to pulp-python to prevent the same problem. pulp/pulp_python#1163

One of things I was concerned about was the performance of trying to validate the modify call, at request time, will not produce an overwrite against the potential thousands of units a user can call with. Have you tested this against your own instance? How large are the modify calls you typically use?

[daviddavis]

@gerrod3 very interesting. I haven't tested this in our instance yet--we actually validate requests and check them for conflicts/substitutions before passing them to Pulp. We do this using the Pulp API so as you can imagine it's slow and not a great solution.

Performance has been an issue with our current solution but luckily most of our users are only adding one or two packages to a repo at a time. I think adding 30 packages is the biggest request size we see.

Your solution looks really clean. I wonder if it could be moved into pulpcore? The one issue though that would prevent us from being able to adopt it is that our repos are shared by multiple teams/users. So if two users are adding packages to a repo around the same timeframe, one user might want to overwrite packages while the other might not. Perhaps one solution might be to also have a allow_package_substitution parameter for repo modify and package upload that override the repo field?

[dkliban]

@daviddavis I am concerned that doing the validation in the API is going to cause timeouts. How do you feel about performing the validation in the task?

[daviddavis]

@dkliban moved the validation to the tasks. I perform the check at the beginning of the task, but it could in theory go into the finalize repo version step. I figure checking early fails fast and avoids wasted work building a version that will be discarded.