Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for token prefix masking #20

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

Add support for token prefix masking #20

wants to merge 9 commits into from

Commits on Nov 25, 2022

  1. Support masking directly in the gitlab-runner-rs crate 

    This brings in a dependency on the mask crate. The code to support
masking artifacts will come later. This only masks the logs passed to
Gitlab.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    fdc0b69 View commit details
    Browse the repository at this point in the history

  2. Add a test for log masking 

    This is only a basic test that the masking is in fact being applied to
variables marked as masked. This is not a test of the functionality of
the masking crate, only of its integration into this crate.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    6cd41e5 View commit details
    Browse the repository at this point in the history

  3. Allow client crates to upload masked files 

    Although we already automatically mask the trace sent to Gitlab, some
runners will produce additional log files or metadata, which must also
be masked to prevent secrets from being revealed in the artifact data.

This permits client crates to identify a file they wish to upload as
additionally requiring masking.

Note that the version bump in the masker crate is because we now need
to be able to clone the masker for each Uploader to use.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    e4d8dce View commit details
    Browse the repository at this point in the history

  4. Parse GitlabFeatures from the JobResponse 

    This is the other half of the capability signalling that Gitlab does.
Not only does a runner signal what features it is capable of supporting
when making a request, but the server also provides some information
back to the runner about what capabilities it provides.

There are currently three capabilities Gitlab can indicate it has
(this commit only adds parsing not support):

- Separating the trace into sections for easier viewing.
- Prefix masking (it's not totally clear why this is a Gitlab feature -
  but this is the mechanism by which the prefix masks are communicated to
  a runner willing to mask them).
- Failure reasons.

The end goal here is support for prefix masking, so that things like
PAT tokens are all automatically masked system-wide in logs.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    1ea6feb View commit details
    Browse the repository at this point in the history

  5. Enable token prefix masking 

    If the features passed by the Gitlab server indicate that there are
token prefixes that should be masked, this adds those prefixes to the
Masker. This is fully transparent to users of this crate; the correct
thing will happen provided they correctly identify which artifacts
they believe should be masked. The trace will always be masked
correctly.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    f8865ba View commit details
    Browse the repository at this point in the history

  6. Add support for token prefix masking in the mock 

    Allow token prefixes to be registered on the mock job. If they
are present, add them to the Gitlab features provided by the mock
with that job, as Gitlab itself does.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    e7288ea View commit details
    Browse the repository at this point in the history

  7. Allow job artifacts to be retrieved from the mock 

    This is useful when testing whether the artifacts uploaded are as
expected.  In particular, we need this now to test whether masked
artifacts are actually being masked as we expect them to be.

Note that we are returning the raw binary of the zipfile, and we'll
need to use the zipfile crate to actually interpret this data. That
seems acceptable for now.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    6c79602 View commit details
    Browse the repository at this point in the history

  8. Use less common token values for masked variables 

    Using very common short names for tokens like 'token', when this is
also masked, can now have odd effects on other tests, because we may
unexpectedly mask out these words from log data. It's safer to use
slightly longer stand-ins. In particular, they conflict with Gitlab's
own tests of prefix matching.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    fda0bc2 View commit details
    Browse the repository at this point in the history

  9. Add the gitlab prefix masking tests directly 

    This uses the actual source code of those tests, which is MIT licensed
(as is this crate), to be certain that we match the behaviour of the
upstream runner.
    @eds-collabora
    eds-collabora committed Nov 25, 2022
    Configuration menu
    Copy the full SHA
    42e38e0 View commit details
    Browse the repository at this point in the history