contest: retry when unable to load input files

[matttbe]

The contest service was unable to load the 'branch_info' input file twice yesterday. Probably because it was being updated:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Catch the error, and retry max 4 more times with the same delay.

[kuba-moo]

I thought we had some "atomic update" method for updating the JSON files. Does it still race? Maybe some files are missing using that atomic update API?

[matttbe]

From what I understood, nipa-br-faker.service is executed every 2 minutes, and it combines branches-info-nn.json and branches-info-hw.json into branches-info.json used by the "contest" service. The faker service reads the 2 json, combines them, opens the output file, truncating the file first, then writes it and closes it:

nipa/contest/results-faker.py

Lines 26 to 37 in 95a2926

	def combine_infos(config):
	paths = config.get("input", "infos", fallback="").split(',')
	if not paths:
	return
	infos = {}
	for path in paths:
	with open(path, "r") as fp:
	infos.update(json.load(fp))
	with open(config.get("output", "info"), 'w') as fp:
	json.dump(infos, fp)

It looks like there is a very small window where the file can be empty.

[kuba-moo]

That's the one, it should be doing an "atomic" write, write to a temp file and rename it overriding the existing file, instead of dumping directly to the output.