Option to delay Freight verification in a Stage

[jessesuen]

Proposed Feature

As soon as a Freight is successfully promoted into a Stage, it is immediately verified to go to downstream Stages. This is often too fast for users. Instead, they wish for there to be a minimum "soak" or "bake" time before it can automatically go downstream (e.g. 1h, 1d).

We could introduce a verificationDelay option in a Stage where a Freight must be running in a Stage for some minimum amount of time, before it is qualified to downstream

Motivation

Users want automated promotions, but don't want for it to be immediate.

Suggested Implementation

Delay the marking of a Freight to be verifiedIn a stage until it passes the user specified time.

There is a workaround for now, which is to reference an AnalysisTemplate that has an initialDelay. e.g.:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: soak
spec:
  metrics:
    - name: soak
      count: 1
      initialDelay: 1h
      provider:
        web:
          url: https://example.com # Choose a URL that will return 200