This is an application to summarize the variables in a data frame. It will accept a CSV, TSV or XLS file and produce a table summarizing all columns individually.
This was motivated by the fact that the summary function for a pandas data frame ignores all non-numeric columns, and does not contain multiple common analytical considerations: how many unique values, how many missing values, min and max dates, min, mean and max string lengths.
Output can be generated as either Latex or Markdown.
Released and distributed via setuptools/PyPI/pip for Python 3.
Additional detail available in the companion blog post
Initial implementation can handle larger files by chunking data and iteratively building statistics. All statistics are robust except for estimation of the proportion of unique values. We have used a simple implementation of the Flajolet Martin algorithm based on the implementation by Javia Jinkal
This review article by Phillip Gibbons gives a great overview of the alternatives.
You can use this application multiple ways
Use the runner:
./dfsummarizer-runner.py markdown data/test.csv > markdown_test.md
Which was used to generate the markdown output test file
Invoke the directory as a package:
python -m dfsummarizer markdown data/test.csv
Or simply install the package and use the command line application directly
Installation from the source tree:
python setup.py install
(or via pip from PyPI):
pip install dfsummarizer
Now, the dfsummarizer command is available::
dfsummarizer markdown test.csv
This will produce a markdown table summarizing the contents of the CSV file test.csv
Python package built using the bootstrap cmdline template by jgehrcke