15. Stingray Build

The source for stingray is a Sphinx project that depends on PyLit. Yes. The documentation spawns the code.

Note

PyLit Feature

The first line of each file should be ..    #!/usr/bin/env python3.

The four spaces between .. and #! defines the indent used for each line of code in the file.

Four spaces.

Stingray depends on the following

In addition to Python 3.3, there are two other projects used to build.

The PyLit install is little more than a download and move the pylit.py file to the Python site-packages directory.

Sphinx and XLRD should be installed with easy_install.

easy_install xlrd
easy_install sphinx

In the case of having Python2 and Python3 installed, easy_install-3.3 may be required. On most systems, sudo is also required.

The diagrams are done with YUML. See http://yuml.me.

15.1. Build Procedure

  1. Bootstrap the build.py script by running PyLit.

    python3 -m pylit -t source/build.rst build.py

    This reports that an extract was written to build.py.

  2. Use the build.py script to create the stingray source, unit tests, demonstration applications. Build the Sphinx documentation. And run the unit test, too.

    python3 build.py

    At the end of this step, the directory tree will include the following.

    • build. The documentation. In HTML.
    • demo. Some demonstration applications that use stingray. See Stingray Demo Applications.
    • stingray. The Python library, ready for installation.
    • test. The unit test script.

    This reports, also, that 174 tests were run.

In general (i.e., any OS except Windows), it’s sensible to do this:

chmod +x build.py

This allows us to use the following for a rebuild:

./build.py

15.2. Build Script Design

This is a platform-independent build.py file for the build script. This can use from sphinx.application import Sphinx and import pylit to access these modules from within Python instead of using command-line shell script techniques.

15.2.1. Overheads

We’re going to make use of three “applications”.

  • Sphinx top-level application.
  • PyLit top-level application.
  • Unittest top-level test runner.
"""Platform-independent build script"""
from __future__ import print_function
import os
import sys
import errno
from sphinx.application import Sphinx
import pylit
import unittest
import logging
import shutil

15.2.2. Sphinx Build

sphinx_build(srcdir, outdir, buildername='html')

Handle the simple use case for the sphinx-build script.

def sphinx_build( srcdir, outdir, buildername='html' ):
    """Essentially: ``sphinx-build $* -b html source build/html``"""
    confdir= srcdir= os.path.abspath( srcdir )
    outdir= os.path.abspath( outdir )
    doctreedir = os.path.join(outdir, '.doctrees')
    app = Sphinx(srcdir, confdir, outdir, doctreedir, buildername)
    app.build(force_all=False, filenames=[])
    return app.statuscode

15.2.3. PyLit Build

pylit_build(srcdir, outdir)

Handle the simple use case for PyLit.

This also handles the necessary rewrite to modify standard paths to Windows paths.

def pylit_build( infile, outfile ):
    """Essentially: ``python3 -m pylit -t source/demo/data_quality.rst demo/test.py``

    The issue here is that we need to provide platform-specific paths.
    """
    if os.sep != '/':
        # Fix windows paths.
        infile= os.path.join( *infile.split('/') )
        outfile= os.path.join( *outfile.split('/') )
    pylit.main( txt2code= True, overwrite="yes", infile= infile, outfile= outfile )

15.2.4. Make Directories

mkdir(path)

Handles the simple use case for assuring that the directory tree exists.

This also handles a rewrite to modify standard paths to Windows paths.

def mkdir( path ):
    if os.sep != '/':
        # Fix windows paths.
        path= os.path.join( *path.split('/') )
    try:
        os.makedirs( path )
    except OSError as e:
        if e.errno == errno.EEXIST:
            pass
        else:
            raise

15.2.5. Copy Data File

copy_file(srcdir, outdir)

Handles the simple use case for copying a file.

def copy_file( srcdir, outdir ):
    """Essentially: ``cp srcdir outdir``"""
    shutil.copy2( srcdir, outdir )

15.2.6. Run the Test Script

run_test()

In effect, this does python3 test/main.py

def run_test():
    import test.main
    result= test.main.main()
    if result.failures:
        sys.exit(result.failures)

15.2.7. The Build Sequence

def build():
    mkdir( 'stingray/schema' )
    mkdir( 'stingray/cobol' )
    mkdir( 'stingray/workbook' )

    pylit_build( 'source/stingray_init.rst', 'stingray/__init__.py' )
    pylit_build( 'source/cell.rst', 'stingray/cell.py' )
    pylit_build( 'source/sheet.rst', 'stingray/sheet.py' )
    pylit_build( 'source/workbook/init.rst', 'stingray/workbook/__init__.py' )
    pylit_build( 'source/workbook/base.rst', 'stingray/workbook/base.py' )
    pylit_build( 'source/workbook/csv.rst', 'stingray/workbook/csv.py' )
    pylit_build( 'source/workbook/xls.rst', 'stingray/workbook/xls.py' )
    pylit_build( 'source/workbook/xlsx.rst', 'stingray/workbook/xlsx.py' )
    pylit_build( 'source/workbook/ods.rst', 'stingray/workbook/ods.py' )
    pylit_build( 'source/workbook/numbers_09.rst', 'stingray/workbook/numbers_09.py' )
    pylit_build( 'source/workbook/numbers_13.rst', 'stingray/workbook/numbers_13.py' )
    pylit_build( 'source/workbook/fixed.rst', 'stingray/workbook/fixed.py' )
    pylit_build( 'source/schema.rst', 'stingray/schema/__init__.py' )
    pylit_build( 'source/schema_loader.rst', 'stingray/schema/loader.py' )
    pylit_build( 'source/cobol_init.rst', 'stingray/cobol/__init__.py' )
    pylit_build( 'source/cobol_loader.rst', 'stingray/cobol/loader.py' )
    pylit_build( 'source/cobol_defs.rst', 'stingray/cobol/defs.py' )
    pylit_build( 'source/snappy.rst', 'stingray/snappy.py' )
    pylit_build( 'source/protobuf.rst', 'stingray/protobuf.py' )
    pylit_build( 'source/installation.rst', 'setup.py' )

    copy_file( 'source/Numbers.json', 'stingray/Numbers.json' )
    copy_file( 'source/Common.json', 'stingray/Common.json' )

    mkdir( 'test' )

    pylit_build( 'source/testing/test_init.rst', 'test/__init__.py' )
    pylit_build( 'source/testing/main.rst', 'test/main.py' )
    pylit_build( 'source/testing/cell.rst', 'test/cell.py' )
    pylit_build( 'source/testing/sheet.rst', 'test/sheet.py' )
    pylit_build( 'source/testing/schema.rst', 'test/schema.py' )
    pylit_build( 'source/testing/schema_loader.rst', 'test/schema_loader.py' )
    pylit_build( 'source/testing/workbook.rst', 'test/workbook.py' )
    pylit_build( 'source/testing/cobol.rst', 'test/cobol.py' )
    pylit_build( 'source/testing/cobol_loader.rst', 'test/cobol_loader.py' )
    pylit_build( 'source/testing/cobol_2.rst', 'test/cobol_2.py' )
    pylit_build( 'source/testing/snappy_protobuf.rst', 'test/snappy_protobuf.py' )

    mkdir( 'demo' )

    pylit_build( 'source/demo/data_quality.rst', 'demo/test.py' )
    pylit_build( 'source/demo/validation.rst', 'demo/app.py' )
    pylit_build( 'source/demo/profile.rst', 'demo/profile.py' )
    pylit_build( 'source/demo/cobol_reader.rst', 'demo/cobol_reader.py' )

    run_test()

    sphinx_build( 'source', 'build/html', 'html' )
    sphinx_build( 'source', 'build/latex', 'latex' )

15.2.8. Main Program Switch

When the build.py script is run from the command line, it will execute the build() function. When it is imported, however, it will do nothing special.

if __name__ == "__main__":
    build()

15.3. Additional Builds

Sometimes it’s desriable to refresh the documentation.

The HTML pages are built with this command.

sphinx-build $* -b html source build/html

The LaTeX document is built with this command.

sphinx-build $* -b latex source build/latex