Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Faulty behaviour when testing documentation for versioned models #28

Open
matteoannotell opened this issue Jul 31, 2023 · 0 comments
Open

Comments

@matteoannotell
Copy link

matteoannotell commented Jul 31, 2023

Problem:

When having a model with multiple versions, the code will disregard this fact, and verify columns against the base schema, not the version specific. This will result in false positive whenever one has for example a base model with a certain amount of columns, and then version(s) defined by excluding certain columns.
Example:

 - name: production_time
    description: |
      A table containing stuff
    latest_version: 1
    config:
      contract: {enforced: true}
    meta:
      owner: data-enablement
    columns:
      - name: last_assignment_id_in_chain
        description: The last assignmentId of chain `chain_id` (currently this can change).
        data_type: integer
      - name: last_completed
        description: The date the last assignement in the assignment chain was completed
        data_type: datetime
      - name: number_of_points
        description: Number of points
        data_type: integer
      - name: number_of_3d_cube_properties
        description: Number of 3D Cube properties
        data_type: integer
      - name: number_of_polygon_properties
        description: Number of polygon properties
        data_type: integer
    versions:
      - v: 1
      - v: 2
        tests:
          - test_versions_equal:
              old_version: ref('production_time', v='1')
              temporal_dimension: last_completed
        columns:
          - include: all
            exclude: [
              number_of_3d_cube_properties,
              number_of_polygon_properties
            ]

results in:

13:58:26  Encountered an error while running operation: Compilation Error in macro required_docs (macros/required_docs.sql)
  The following columns are missing from the model yml:
   - production_time.number_of_3d_cube_properties
   - production_time.number_of_polygon_properties
  
  > in macro default__format_raise_error (macros/utils/formatters/format_raise_error.sql)
  > called by macro format_raise_error (macros/utils/formatters/format_raise_error.sql)
  > called by macro default__required_docs (macros/required_docs.sql)
  > called by macro required_docs (macros/required_docs.sql)
  > called by macro required_docs (macros/required_docs.sql)

I suspect changing this line:

{% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name))

with

 {% set model_columns = adapter.get_columns_in_relation(ref(model.package_name, model.name, version=model.version))

could solve the problem.
It shouldn't (to be verified) cause any back-compatibility problems with dbt-core<1.5 as the extra kwarg should (to be verified) ignored by ref()

Thanks a lot for the good work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant