Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Artifactory Query Language integration #72

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ This library enables you to manage Artifactory resources such as users, groups,
+ [Copy artifact to a new location](#copy-artifact-to-a-new-location)
+ [Move artifact to a new location](#move-artifact-to-a-new-location)
+ [Delete an artifact](#delete-an-artifact)
* [Artifactory Query Language](#artifactory-query-language)
* [Contributing](#contributing)

<!-- tocstop -->
Expand Down Expand Up @@ -320,6 +321,26 @@ artifact = art.artifacts.move("<CURRENT_ARTIFACT_PATH_IN_ARTIFACTORY>","<NEW_ART
art.artifacts.delete("<ARTIFACT_PATH_IN_ARTIFACTORY>")
```

### Artifactory Query Language
You can use [Artifactory Query Language](https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language) to uncover data related to the artifacts and builds stored within Artifactory
```python
from pyartifactory import Artifactory
from pyartifactory.models import Aql

art = Artifactory(url="ARTIFACTORY_URL", auth=('USERNAME','PASSWORD_OR_API_KEY'))

# Create an Aql object with your query parameters
aql_obj = Aql(**{
"domain":"items",
"find":{"name" : {"$match":"*.jar"}},
"sort": { "$asc" : ["repo","name"] },
"limit": 100
})

result = art.aql.query(aql_obj)
>> print(result)
[{'repo': 'my-repo', 'path': 'my/path', 'name': 'test.jar', 'type': 'file', 'size': 1111, 'created': 'some-date', 'created_by': 'some-date', 'modified': 'some-data', 'modified_by': 'some-user', 'updated': 'some-data'}]
```

### Contributing
Please read the [Development - Contributing](./CONTRIBUTING.md) guidelines.
1 change: 1 addition & 0 deletions pyartifactory/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
ArtifactoryRepository,
ArtifactoryArtifact,
ArtifactoryPermission,
ArtifactoryAql,
Artifactory,
AccessTokenModel,
)
Expand Down
4 changes: 4 additions & 0 deletions pyartifactory/exception.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,7 @@ class PropertyNotFoundException(ArtifactoryException):

class InvalidTokenDataException(ArtifactoryException):
"""The token contains invalid data."""


class AqlException(ArtifactoryException):
"""AQL search failed"""
1 change: 1 addition & 0 deletions pyartifactory/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SimpleRepository,
)

from .aql import Aql
from .artifact import (
ArtifactPropertiesResponse,
ArtifactStatsResponse,
Expand Down
28 changes: 28 additions & 0 deletions pyartifactory/models/aql.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
"Artifactory queries"
from typing import List, Dict, Optional, Union, Any
from enum import Enum

from pydantic import BaseModel


class SortTypesEnum(str, Enum):
"Order of query results"
asc = "$asc"
desc = "$desc"


class DomainQueryEnum(str, Enum):
"Artifactory domain objects to be queried"
items = "items"
builds = "builds"
entries = "entries"
Comment on lines +14 to +18
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly this part of the docs

AQL is constructed as a set of interconnected domains as displayed in the diagram below. You may run queries only on one domain at a time, and this is referred to as the Primary domain of the query.
Currently, the following are supported as primary domains: Item, Build, Entry, Promotion and Release. i.e., your queries may be of the form: items.find(...), builds.find(...), archive.entries.find(...), build.promotions.find(...) or releases.find(...).

Artifactory actually supports more than just the "items, builds or entries" that they announce. And even more, you cannot just use entries.find() (or at least on our instance it returns "Failedtoparsequery").

FYI I tested all the primary domains they advertise in the quoted docs on our instance, and none of them fail (we have no results for releases and build.promotions but that may be because we never used the feature)



class Aql(BaseModel):
"Artifactory Query Language"
domain: DomainQueryEnum = DomainQueryEnum.items
find: Optional[Dict[str, Union[str, List[Dict[str, Any]], Dict[str, str]]]]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if you could make a self-referential type here...
The "find" criterion can be a Dict of:

  • strings
  • dict of strings
  • list of find criteria

But after thinking about it I'm not sure you could extract such a type with forward references, because it's not a Pydantic model but a union of many types...
Maybe something like this could work:

AqlFindCriterion = Dict[str, Union[str, Dict[str, str], List["AqlFindCriterion"]]]

include: Optional[List[str]]
sort: Optional[Dict[SortTypesEnum, List[str]]] = None
offset: Optional[int] = None
limit: Optional[int] = None
54 changes: 54 additions & 0 deletions pyartifactory/objects.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""
Definition of all artifactory objects.
"""
import json
import warnings
import logging
import os
Expand All @@ -24,6 +25,7 @@
PermissionNotFoundException,
InvalidTokenDataException,
PropertyNotFoundException,
AqlException,
ArtifactNotFoundException,
)

Expand All @@ -48,6 +50,7 @@
User,
Permission,
SimplePermission,
Aql,
ArtifactPropertiesResponse,
ArtifactStatsResponse,
ArtifactInfoResponse,
Expand All @@ -73,6 +76,7 @@ def __init__(
self.repositories = ArtifactoryRepository(self.artifactory)
self.artifacts = ArtifactoryArtifact(self.artifactory)
self.permissions = ArtifactoryPermission(self.artifactory)
self.aql = ArtifactoryAql(self.artifactory)


class ArtifactoryObject:
Expand Down Expand Up @@ -950,3 +954,53 @@ def delete(self, artifact_path: str) -> None:
artifact_path = artifact_path.lstrip("/")
self._delete(f"{artifact_path}")
logger.debug("Artifact %s successfully deleted", artifact_path)


def create_aql_query(aql_object: Aql):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this function could become a method in the Aql class, to be used something like this:

aql = Aql(<params>)
aql.to_query()  # items.find().include()...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also I think this method could benefit from being more unit tested, to make sure we generate the correct query whatever the combination of parameters we give. I can write those if you want 😉

"Create Artifactory query"
aql_query_text = f"{aql_object.domain}.find"

if aql_object.find:
aql_query_text += f"({json.dumps(aql_object.find)})"
else:
aql_query_text += "()"
Comment on lines +961 to +966
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep it consistent with the other parts of the function (and simpler to understand), maybe move the .find to the if block:

Suggested change
aql_query_text = f"{aql_object.domain}.find"
if aql_object.find:
aql_query_text += f"({json.dumps(aql_object.find)})"
else:
aql_query_text += "()"
aql_query_text = f"{aql_object.domain}"
if aql_object.find:
aql_query_text += f".find({json.dumps(aql_object.find)})"
else:
aql_query_text += ".find()"


if aql_object.include:
format_include = (
json.dumps(aql_object.include).replace("[", "").replace("]", "")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this, what would happen if there is a [ in one of the included fields? It would probably not happen, I know, but maybe just removing them as the first and last characters would be safer?

)
aql_query_text += f".include({format_include})"

if aql_object.sort:
sort_key = list(aql_object.sort.keys())[0]
sort_value = json.dumps(aql_object.sort[sort_key])
aql_query_text += f'.sort({{"{sort_key.value}": {sort_value}}})'

if aql_object.offset:
aql_query_text += f".offset({aql_object.offset})"

if aql_object.limit:
aql_query_text += f".limit({aql_object.limit})"
Comment on lines +979 to +983
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be careful with this sort of comparison, if something is not the same as if something is not None! In the first case you will not enter the if if the user sends a 0!


return aql_query_text


class ArtifactoryAql(ArtifactoryObject):
"Artifactory Query Language support"
_uri = "search/aql"

def query(self, aql_object: Aql) -> List[Dict[str, Union[str, List]]]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then I wondered about the API we expose... Should the user create an Aql object, and pass it to the method, or should the method take parameters like find, include... that would individually be typed?
I don't really know the purpose of the Aql object, but maybe you thought about it and have a reason that I don't know ^^

"Send Artifactory query"
aql_query = create_aql_query(aql_object)
try:
response = self._post(f"api/{self._uri}", data=aql_query)
response_content: List[Dict[str, Union[str, List]]] = response.json()[
"results"
]
Comment on lines +997 to +999
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should parse the response here, with parse_from_obj(List[Dict[str, Union[str, List]]], response.json()["results"] (and you won't have to type it because parse_from_obj is cleverly typed 😉)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, actually should we return only the results key? The user might need the other keys, for example to know how many results were returned.

logging.debug("Successful query")
return response_content
except requests.exceptions.HTTPError as error:
raise AqlException(
"Bad Aql Query: please check your parameters."
"Doc: https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language"
) from error
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ tests

[tool.pylint.messages_control]
disable = """
bad-continuation,line-too-long,too-few-public-methods,import-error,too-many-instance-attributes
bad-continuation,line-too-long,too-few-public-methods,import-error,too-many-instance-attributes,too-many-lines
"""

[tool.pylint.basic]
Expand Down
55 changes: 55 additions & 0 deletions tests/test_aql.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import pytest
import responses

from pyartifactory import ArtifactoryAql
from pyartifactory.exception import AqlException
from pyartifactory.models import AuthModel, Aql

URL = "http://localhost:8080/artifactory"
AUTH = ("user", "password_or_apiKey")
AQL_RESPONSE = {
"results": [
{
"repo": "libs-release-local",
"path": "org/jfrog/artifactory",
"name": "artifactory.war",
"type": "item type",
"size": "75500000",
"created": "2015-01-01T10:10;10",
"created_by": "Jfrog",
"modified": "2015-01-01T10:10;10",
"modified_by": "Jfrog",
"updated": "2015-01-01T10:10;10",
}
],
"range": {"start_pos": 0, "end_pos": 1, "total": 1},
}


@responses.activate
def test_aql_success():
responses.add(
responses.POST, f"{URL}/api/search/aql", json=AQL_RESPONSE, status=200
)

artifactory_aql = ArtifactoryAql(AuthModel(url=URL, auth=AUTH))
aql_obj = Aql(**{"find": {"repo": {"$eq": "libs-release-local"}}})
result = artifactory_aql.query(aql_obj)
assert result == AQL_RESPONSE["results"]


@responses.activate
def test_aql_fail_baq_query():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wooops, that should read ^^

Suggested change
def test_aql_fail_baq_query():
def test_aql_fail_bad_query():

responses.add(
responses.POST, f"{URL}/api/search/aql", json=AQL_RESPONSE, status=400
)

artifactory_aql = ArtifactoryAql(AuthModel(url=URL, auth=AUTH))
aql_obj = Aql(
include=["artifact", "artifact.module", "artifact.module.build"],
sort={"$asc": ["remote_downloaded"]},
limit=100,
)

with pytest.raises(AqlException):
artifactory_aql.query(aql_obj)