IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLS, XML, and JSON.
from itertable import load_file
for row in load_file("example.xls"):
print(row.date, row.name)
Note: Prior to version 2.0, IterTable was wq.io, a submodule of the wq framework. The package has been renamed to avoid confusion with the wq framework website (https://wq.io). Similarly, IterTable's
*IO
classes have been renamed to*Iter
, as the API is not intended to match that of Python'sStringIO
or otherio
classes.
- from wq.io import CsvFileIO
- data = CsvFileIO(filename='data.csv')
+ from itertable import CsvFileIter
+ data = CsvFileIter(filename='data.csv')
# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate
python3 -m pip install itertable
# GIS support (Fiona & Shapely)
python3 -m pip install itertable[gis]
# Excel write support
python3 -m pip install itertable[write]
# (xls/xlsx read support is enabled by default)
# Pandas integration
python3 -m pip install itertable[pandas]
IterTable provides a general purpose API for loading, iterating over, and writing tabular datasets. The goal is to avoid needing to remember the unique usage of e.g. csv, xlrd, or xml.etree every time one needs to work with external data. Instead, IterTable abstracts these libraries into a consistent interface that works as an iterable of namedtuples. Whenever possible, the field names for a dataset are automatically determined from the source file, e.g. the column headers in an Excel spreadsheet.
from itertable import ExcelFileIter
data = ExcelFileIter(filename='example.xls')
for row in data:
print(row.name, row.date)
IterTable provides a number of built-in classes like the above, including a CsvFileIter
, XmlFileIter
, and JsonFileIter
. There is also a convenience function, load_file()
, that attempts to automatically determine which class to use for a given file.
from itertable import load_file
data = load_file('example.csv')
for row in data:
print(row.name, row.date)
All of the included *FileIter
classes support both reading and writing to external files, though write support for Excel files requires itertable[write]
(which installs xlwt
and xlswriter
).
IterTable also provides network-capable equivalents of each of the above classes, to facilitate loading data from third party webservices.
from itertable import JsonNetIter
class WebServiceIter(JsonNetIter):
url = "http://example.com/api"
data = WebServiceIter(params={'type': 'all'})
for row in data:
print(row.timestamp, row.value)
The powerful requests library is used internally to load data over HTTP.
When Pandas is installed (via itertable[pandas]
), the as_dataframe()
method on itertable classes can be used to create a DataFrame, enabling more extensive analysis possibilities.
instance = WebServiceIter(params={'type': 'all'})
df = instance.as_dataframe()
print(df.value.mean())
When Fiona and Shapely are installed (via itertable[gis]
), itertable can also open and create shapefiles and other OGR-compatible geographic data formats.
from itertable import ShapeIter
data = ShapeIter(filename='sites.shp')
for id, site in data.items():
print(id, site.geometry.wkt)
More information on IterTable's gis support is available here.
It is straightforward to extend IterTable to support arbitrary formats. Each provided class is composed of a BaseIter class and mixin classes (loaders, parsers, and mappers) that handle the various steps of the process.
© 2013-2019 by S. Andrew Sheppard