Skip to content

Commit

Permalink
Migrate repo to sfc
Browse files Browse the repository at this point in the history
  • Loading branch information
scaleoutsean committed Apr 27, 2024
0 parents commit b903fbc
Show file tree
Hide file tree
Showing 38 changed files with 12,784 additions and 0 deletions.
167 changes: 167 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
cleanup.sh
vmwcollector/vsphere-graphite.json
*/__pycache__/

### GH for Python

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

97 changes: 97 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Change Log

- [Change Log](#change-log)
- [Changes in v0.7.1](#changes-in-v071)
- [Changes in v0.7](#changes-in-v07)
- [Changes in v0.6.1](#changes-in-v061)
- [Changes in .v6](#changes-in-v6)
- [Changes in .5](#changes-in-5)
- [Changes in .4](#changes-in-4)
- [Changes in .3](#changes-in-3)
- [Changes in .v2](#changes-in-v2)
- [Changes in .v1](#changes-in-v1)

## Changes in v0.7.1

- Create SolidFire container based on v0.7 for easy deployment and stand-alone SolidFire monitoring with Kubernetes (see ./sfcollector-kubernetes/ directory)
- Update README.md and add an example Deployment YAML file for SolidFire Collector and Graphite/statsd
- SolidFire Collector container published on Docker Hub can be deployed without any configuration wizard

## Changes in v0.7

- Fork upstream hcicollector by jedimt (this release builds upon upstream branch .v7, here renamed to v0.7)
- Remove NetApp Trident-related steps from install script (see the FAQs). HCICollector now by default uses two local Docker volumes: one for GraphiteDB and another for Grafana settings
- Remove the NetApp Technical Report PDF and video demo files from the repo for faster repository cloning. Add video links to YouTube demo videos
- Changes and improvements to documentation as well as online help (links to the SolidFire UI and basic descriptions in various panels)
- Introduce potentially breaking changes in metrics paths and details gathered from SolidFire (see Release Notes v0.7 and FAQs)
- Change storage schemas for GraphiteDB to use less disk space
- Fixes:
- SFCollector: wrapper script can contain special characters (issue #2). Change Docker base OS to Python 3.8.6 (slim Buster)
- SFCollector: gather more SolidFire metrics relevant to administrators and operations staff
- SFCollector: deduplication efficiency formula changed to account for space used for snapshots (issue #3)
- SFCollector: set SolidFire API call timeout to a lower value than default (issue #6)
- SFCollector: add the option to validate TLS certificate of SolidFire API endpoint(s)
- SFCollector: add variable for API response timeout for larger environments (issue #12)
- SFCollector: upgrade SolidFire SDK for Python v1.7.0.152, upgrade base image to Python 3.9.2-buster-slim
- Grafana: configure Legend and Axis Y values in most panels to display 0 decimals (enforce integer values where apppropriate (e.g. byte count) and lower the level of unnecessary detail elsewhere), adjust precision and make other usability improvements
- Grafana: change deprecated gauge caunters to new gauge counters
- Grafana: replace deprecated Grafana renderer with new renderer container
- Grafana: add new panels to existing dashboards, including iSCSI connections, disk wear level, QoS histograms and more
- Grafana: change some dashboards to make it easier to see key panels without scrolling
- Update third party container images:
- graphite-statsd v1.1.7-11: considerably smaller Docker image, and (by upstream) update internal components (`GRAPHITE_LOG_ROTATION: 1` added to container environment variables to retain previous behavior after changes in upstream v1.1.7-9)
- grafana v6.7.5: v6.7.4 and v6.7.5 fix two security issues (neither of them impacts HCICollector with default settings because they aren't enabled)
- vsphere-graphite v0.8b: support for vSphere 7.0 (and 7.0U1) API via govmomi 0.23.0 (v7.0 API should work fine on vCenter 7.0U1)
- Known issues:
- Built-in dashboard links to SolidFire UI work for configurations with single SolidFire storage cluster. HCICollector environments that monitor multiple SolidFire clusters can add a MVIP variable to dashboard and reference it in URLs to modify URLs on the fly
- Install script configures only one vCenter cluster and only one SolidFire cluster. See the FAQs for workarounds
- Some visualizations use Beta-release plugins from Grafana which may have issues related to visualization or configuration (editing of panel settings). There are bugs in browsers and Grafana too
- SolidFire disk drive Wear Level / Life Remaining visualization uses the title like `drive.${disk-id}` although `${disk-id}` would be better. The reason is the Grafana plugin cannot accept numeric titles. If that bothers you try some other Grafana visualization plugin
- Dashboards and panels may contain hard-coded URLs (e.g. 192.168.1.30) or SolidFire cluster name (e.g. PROD): search-and-replace this link with your own before you import them. HCICollector install script does this for you, but direct import bypasses that step. The proper solution would be to add the MVIP variable to all dashboards and use it in URLs
- By the nature of how vSphere and SolidFire plugins gather metrics, an object deleted and created with the same name (e.g. a VM, or datastore, or SolidFire volume) may appear in the same long term graph as its ancient namesake. Such objects generally have UUIDs, but VMware and SolidFire don't keep UUID-to-Name mappings for deleted objects. If UUIDs were to be used (which is partially done in Some SolidFire dashboards, that would be correct but present problems for humans who hardly recognize current, let alone historic, UUIDs). This isn't an unsolvable problem but it'd require a fair amount of work to fix. Essentially a DB would have to be built, and another back-end added to Graphite, as far as I can tell. Or cross reference SolidFire API logs against UUIDs.
- Experimental features:
- Two sample dashboards for hardware monitoring of NetApp H-Series Compute nodes: metrics are not gathered by by default - it requires read-only access to the compute node IPMI interface, manual deployment of collectd VM or container (see the config-examples directory and FAQs) and potentially modifications to the dashboards to make them usable)

## Changes in v0.6.1

- Fix for the bad dedupe factor formula (issue #3) in .v6
- Prior to v0.7, sfcollector used latest version of base OS, so there's a risk to rebuilding containers as base OS updates may break sfcollector
- If you want to try, download branch [v0.6.1](https://github.com/scaleoutsean/hcicollector/tree/v0.6.1) and rebuild, or just apply the [change](https://github.com/jedimt/hcicollector/compare/master...scaleoutsean:v0.6.1) to existing sfcollector/solidfire_graphite_collector.py and rebuild only that container (sfcollector)

## Changes in .v6

- Changed file layout to be more consistent with container names and roles
- Retooled for Grafana 5.0.0
- Dashboards and datasources are now automatically added through the new provisioning functionality in Grafana 5
- Removed the external volume for the Grafana container, only Graphite uses an (optional) external iSCSI volume for persistent data
- Added the ability to poll for active alerts in the "SolidFire Cluster" dashboard.
- Added support for email alerting based on SolidFire events. Note: alerting queries do not support templating variables so if you have multiple clusters you will need to use `*` for the cluster instance instead of the `$Cluster` variable. The net effect of this is that the alert pane will show alerts from ALL clusters instead of an individually selected cluster.
- New detailed install document
- Added a very basic installation script

## Changes in .5

- Extensive dashboard updates. Dashboards now available on [grafana.com](https://grafana.com/dashboards?search=HCI)
- Added additional metrics to collection
- Updated to Trident from NDVP for persistent storage

## Changes in .4

- Added a vSphere collectored based heavily on the work of cblomart's vsphere-graphite collector
- Dashboard updates
- New dashboards for vSphere components

## Changes in .3

- Changed the collector container to Alpine which dramatically cut down container size and build time.
- Other minor changes

## Changes in .v2

- Added "&" in wrapper.sh script to make the collector calls async. Previously the script was waiting for the collector script to finish before continuing the loop. This caused the time between collections to stack which caused holes in the dataset. Now stats should be returned every minute
- Changed graphs to use the summerize function for better accuracy

## Changes in .v1

- Initial release

6 changes: 6 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Contributor Code of Conduct

This project adheres to No Code of Conduct. We accept anyone's contributions. Nothing else matters.

For more information please visit the [No Code of Conduct](https://nocodeofconduct.com) homepage.

Loading

0 comments on commit b903fbc

Please sign in to comment.