Refactor sloc scanner to be a class #124

turbaszek · 2020-12-17T00:14:50Z

This PR:

refactors git sloc scanner as a class
adds loguru for logging purposes
keeps backward compatibility with other scanners

This PR introduces some structure to setup.py by encapsulating some parts of the logic into functions and creating the main function.

* Introduce github actions for CI * fixup! Introduce github actions for CI

Closes: apache#59

Closes: apache#49

This commit refactors the code base to create an installable kibble application. Co-authored-by: Michał Słowikowski <[email protected]>

Add Black to pre-commit and format whole codebase

* Use KibbleConfigParser in setup script * fixup! Use KibbleConfigParser in setup script

…hat.

To check that kibble works we can at least build the docker image, run kibble --help and validate python requirements.

* removed few unused variables from code * bugzilla – removed unused variables from code * git-census – removed unused variable * buildbot – removed unused variable * pipermail – removed unused variable * ponymail – removed unused variable * discourse – removed unused variable * github-issues – removed unused variable * jenkins – removed unused variable * jira – removed unused variable * github-stats – removed unused variable * ponymail – removed unused variable * git-evolution – removed unused variable

Before this commit I got this error when running Kibble locally: This was because https://github.com/apache/kibble/pull/83 replaced kibble.yaml with kibble.ini This PR reads the config from `kibble.ini`. This also adds a network to docker-compose file without which I was getting host not found error.

3.3 is out for https://github.com/pre-commit/pre-commit-hooks

Replace old link with new link

`master` branch was renamed with `main`. This should reflect in doc.

Since https://kibble.apache.org/ is available we should use that instead of http

`Kible` -> `Kibble` `milis` -> `milliseconds` `offlined` -> `offline` `annalyze` -> `analyze` `Verrifying` -> `Verifying` `betwen` -> `between` `fechthing` -> `fetching`

`python:3.8` -- 331.8 MB `python:3.8-slim` -- 41.67 MB https://hub.docker.com/_/python?tab=tags&page=1&ordering=last_updated&name=3.8

This is recommended deviantony/docker-elk#243 (comment) and in official ES docs: https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html

Upgrade black pre-commit from `19.3b0` to `20.8b1`

- Comparison with None - Redundant Parenthesis - Remove unused variable - Fix mutable default arguments - Augmented assignment expressions

…che#111) There was no such variable `ESVersion` -- possibly a typo as `self.ESversion` is defined on line 147

Install cloc (https://github.com/AlDanial/cloc). `cloc` is a Prerequisite as mentioned in https://apache-kibble.readthedocs.io/en/latest/setup.html#installing-the-server Without it the count in the git scanner fails.

This PR adds Github Action job to run tests on CI & fix a failing test

Co-authored-by: Kaxil Naik <[email protected]>

https://docs.github.com/en/free-pro-team@latest/github/building-a-strong-community/configuring-issue-templates-for-your-repository#configuring-the-template-chooser The file name should be `config.yml` not `config.ymal`

The paths were incorrect, I think they are from before when they were in a separate repo

This PR replaces cameleCase with snake_case, removes some unused variables and adds some that in some cases may result in run time error.

This PR moved watson, azure, picoapi and git configuration to kibble.ini config file. It fixes also reading ES configuration. In this way all Kibble configuration is in single place.

This will help us refactor and standarize all scanners. Additionally this class create a good place to have common method which are deuplicated in many places.

turbaszek · 2020-12-17T00:18:14Z

I still got to refactor the logic of running scanner to use the new approach but happy to hear opinions about this approach. @kaxil @michalslowikowski00 @Humbedooh

turbaszek and others added 30 commits October 12, 2020 14:10

Refactor setup.py (apache#48)

a2d61ac

This PR introduces some structure to setup.py by encapsulating some parts of the logic into functions and creating the main function.

Introduce github actions for CI (apache#47)

b371d80

* Introduce github actions for CI * fixup! Introduce github actions for CI

Apply pre-commit checks to all files and make CI green (apache#58)

2002a53

Add Github Action and license badge (apache#60)

9376200

Add and unify license headers (apache#61)

aaae65d

Closes: apache#59

Add dev docker compose (apache#50)

7092e86

Refactor api to be a package (apache#51)

5e18332

Closes: apache#49

Make Kibble a package (apache#67)

edb9a91

This commit refactors the code base to create an installable kibble application. Co-authored-by: Michał Słowikowski <[email protected]>

Add auto-labeler for PRs (apache#68)

340f104

Fix labeler definitions (apache#70)

e4563ca

Add Black to pre-commit (apache#66)

d7f9031

Add Black to pre-commit and format whole codebase

Add kibana dashboard to dokcer-compose (apache#75)

17e6734

Add KibbleConfigParser (apache#74)

2abfcc8

Fix deprecation warning for yaml.load() (apache#82)

4d5f537

Remove ulimits for elasticsearch in docker-compose (apache#80)

5471c07

Use KibbleConfigParser in setup script (apache#83)

6959f3c

* Use KibbleConfigParser in setup script * fixup! Use KibbleConfigParser in setup script

Add scanners to main repository (apache#78)

0a11b51

GitHub uses main as default branch, so we should obviously look for t…

4884cf5

…hat.

Refactor setup scripts as kibble cli command (apache#91)

7d70dd5

Build docker image on CI (apache#92)

3d03a24

To check that kibble works we can at least build the docker image, run kibble --help and validate python requirements.

Add isort pre-commit hook (apache#96)

639f10c

Refactor make_account as cli command (apache#94)

393635e

fix doc typo (apache#99)

5228a4a

Update pre-commits hook from 2.3 to 3.0 (apache#101)

275c55f

3.3 is out for https://github.com/pre-commit/pre-commit-hooks

Fix asf.yaml INFRA link (apache#102)

36e2cf7

Replace old link with new link

Replace mention of 'master' branch with 'main' (apache#103)

d285764

`master` branch was renamed with `main`. This should reflect in doc.

Use https URL for Kibble in setup.py (apache#104)

e07e601

Since https://kibble.apache.org/ is available we should use that instead of http

Fix various typos (apache#105)

076bbe4

`Kible` -> `Kibble` `milis` -> `milliseconds` `offlined` -> `offline` `annalyze` -> `analyze` `Verrifying` -> `Verifying` `betwen` -> `between` `fechthing` -> `fetching`

kaxil and others added 16 commits December 13, 2020 03:08

Use python:3.8-slim image for Dockerfile (apache#107)

f03b7ed

`python:3.8` -- 331.8 MB `python:3.8-slim` -- 41.67 MB https://hub.docker.com/_/python?tab=tags&page=1&ordering=last_updated&name=3.8

Fix ES Memory locking issue (apache#108)

1876559

This is recommended deviantony/docker-elk#243 (comment) and in official ES docs: https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html

Fix error when running Kibble Scanner (apache#109)

d9a3893

Upgrade black to 20.8b1 (apache#112)

729256f

Upgrade black pre-commit from `19.3b0` to `20.8b1`

Improve Code Quality (apache#113)

3485535

- Comparison with None - Redundant Parenthesis - Remove unused variable - Fix mutable default arguments - Augmented assignment expressions

Fix unresolved reference error in kibble/api/plugins/database.py (apa…

c14a053

…che#111) There was no such variable `ESVersion` -- possibly a typo as `self.ESversion` is defined on line 147

Install cloc in Dockerfile to fix git scanner (apache#110)

5bf37a8

Install cloc (https://github.com/AlDanial/cloc). `cloc` is a Prerequisite as mentioned in https://apache-kibble.readthedocs.io/en/latest/setup.html#installing-the-server Without it the count in the git scanner fails.

Fix failing test & Run tests on CI (apache#115)

89748e4

This PR adds Github Action job to run tests on CI & fix a failing test

Show colored output for Pytest & Show diff on pre-commits (apache#116)

ea59ae0

Add scan command to run scanners (apache#114)

f3a4b27

Co-authored-by: Kaxil Naik <[email protected]>

Fix name of config.yml in Github Issue Template (apache#117)

b4025c2

https://docs.github.com/en/free-pro-team@latest/github/building-a-strong-community/configuring-issue-templates-for-your-repository#configuring-the-template-chooser The file name should be `config.yml` not `config.ymal`

Add missing __init__.py file in plugins & cli directory (apache#118)

f042a59

Fix the path for the scanners in scanners/README.md (apache#119)

257992d

The paths were incorrect, I think they are from before when they were in a separate repo

Improve code quality of scanners (apache#120)

f8d731c

This PR replaces cameleCase with snake_case, removes some unused variables and adds some that in some cases may result in run time error.

Make scanners use kibble.ini instead of config.yaml (apache#122)

e19cf6c

This PR moved watson, azure, picoapi and git configuration to kibble.ini config file. It fixes also reading ES configuration. In this way all Kibble configuration is in single place.

Introduce a BaseScanner class (apache#121)

96959ac

This will help us refactor and standarize all scanners. Additionally this class create a good place to have common method which are deuplicated in many places.

turbaszek force-pushed the refactor-sloc-scanner branch from 831656b to 29fdd66 Compare December 17, 2020 00:17

Refactor sloc scanner to be a class

dee60e4

turbaszek force-pushed the refactor-sloc-scanner branch from 29fdd66 to dee60e4 Compare December 17, 2020 00:19

kaxil approved these changes Dec 18, 2020

View reviewed changes

fixup! Refactor sloc scanner to be a class

a3a7a83

github-actions bot added area:cli CLI related issues area:dev labels Dec 18, 2020

turbaszek marked this pull request as ready for review December 18, 2020 18:52

turbaszek force-pushed the main branch 2 times, most recently from 2abfcc8 to 113819c Compare January 16, 2021 11:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor sloc scanner to be a class #124

Refactor sloc scanner to be a class #124

turbaszek commented Dec 17, 2020 •

edited

Loading

turbaszek commented Dec 17, 2020

Refactor sloc scanner to be a class #124

Are you sure you want to change the base?

Refactor sloc scanner to be a class #124

Conversation

turbaszek commented Dec 17, 2020 • edited Loading

turbaszek commented Dec 17, 2020

turbaszek commented Dec 17, 2020 •

edited

Loading