catch failed watch callbacks

[sanoel]

If the watch handler fails, e.g., writes made when syncing with a remote server, we should not commit the change and should retry the callback with an exponential backoff. Implement try/catch around the watch callback, and if caught, print an error message and revert the _rev in the cache.

[tomohiroarakawa]

I like the idea of reverting the cache when the callback throws an exception. I feel that the retry with an exponential backoff should be implemented by the callback but not by oada-cache. There may be a case where we don't want to retry, or we want to retry but with a different backoff parameter, etc...

[aultac]

@tomohiroarakawa if we revert the cache’s rev for the resource, we’ll have to do something to re-emit the change again so the cache can try to catch up. It would make sense for it to re-emit on an exponential backoff. It also would make sense for you to be able to throw something that says to the cache “don’t revert, just forget this change”. I also think it should be an option for the entire watch: i.e. “for this entire watch, here is the default behavior for failed callbacks.” We probably should also not emit newer rev’s until we get a successful handling of the oldest rev or we could get out of order changes, but for @sanoel’s purposes now I think it is fine without dealing with that race condition.