Added: Sphinx docs, testing readthedocs integration

* Added: Sphinx configuration including readthedocs config
 * Added: Sphinx fabric task to auto-document the project
 * Fixed: Some issues in the code causing autodoc to fail when parsing
 * Added: Manual docs - structure, proof of concepts and RsT examples
 * Fixed: RsT formatting in docstrings
 * Fixed: Some adjacent minor style and lint issues

.gitignore

@@ -3,6 +3,7 @@
 **.DS_Store
 src/build
 src/dist
+src/.eggs
 src/.project
 src/.pydevproject
 src/.settings/
@@ -13,3 +14,6 @@ build/lib.*
 build/temp.*
 dist
 *.egg-info
+docs/_*/*
+docs/autodoc/
+pyan/

.travis.yml

@@ -7,6 +7,5 @@ addons:
       - build-essential
       - libcap-dev
 install:
-  - pip install -r requirements.txt
   - python setup.py install
 script: pybitmessage -t

PULL_REQUEST_TEMPLATE.md

@@ -1,15 +1,33 @@
-## Code contributions to the Bitmessage project
+## Repository contributions to the PyBitmessage project
 
-- try to explain what the code is about
-- try to follow [PEP0008](https://www.python.org/dev/peps/pep-0008/)
-- make the pull request against the ["v0.6" branch](https://github.com/Bitmessage/PyBitmessage/tree/v0.6)
-- PGP-sign the commits included in the pull request
-- try to use a good editor that removes trailing whitespace, highlights potential python issues and uses unix line endings
 - You can get paid for merged commits if you register at [Tip4Commit](https://tip4commit.com/github/Bitmessage/PyBitmessage)
 
-If for some reason you don't want to use github, you can submit the patch using Bitmessage to the "bitmessage" chan, or to one of the developers.
+### Code
+
+- Try to refer to github issue tracker or other permanent sources of discussion about the issue.
+- It is clear from the diff *what* you have done, it may be less clear *why* you have done it so explain why this change is necessary rather than what it does
+
+### Documentation
+
+- If there has been a change to the code, there's a good possibility there should be a corresponding change to the documentation 
+- If you can't run `fab build_docs` successfully, ask for someone to run it against your branch
+
+### Tests
+
+- If there has been a change to the code, there's a good possibility there should be a corresponding change to the tests
+- If you can't run `fab tests` successfully, ask for someone to run it against your branch
+
 ## Translations
 
-For helping with translations, please use [Transifex](https://www.transifex.com/bitmessage-project/pybitmessage/). There is no need to submit pull requests for translations.
-For translating technical terms it is recommended to consult the [Microsoft Language Portal](https://www.microsoft.com/Language/en-US/Default.aspx).
+- For helping with translations, please use [Transifex](https://www.transifex.com/bitmessage-project/pybitmessage/).
+- There is no need to submit pull requests for translations.
+- For translating technical terms it is recommended to consult the [Microsoft Language Portal](https://www.microsoft.com/Language/en-US/Default.aspx).
+
+### Gitiquette
+
+- Make the pull request against the ["v0.6" branch](https://github.com/Bitmessage/PyBitmessage/tree/v0.6)
+- PGP-sign the commits included in the pull request
+- Use references to tickets, e.g. `addresses #123` or `fixes #234` in your commit messages
+- Try to use a good editor that removes trailing whitespace, highlights potential python issues and uses unix line endings
+- If for some reason you don't want to use github, you can submit the patch using Bitmessage to the "bitmessage" chan, or to one of the developers.
 

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = PyBitmessage
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,210 @@
+# -*- coding: utf-8 -*-
+"""
+Configuration file for the Sphinx documentation builder.
+
+This file does only contain a selection of the most common options. For a
+full list see the documentation:
+http://www.sphinx-doc.org/en/master/config
+
+-- Path setup --------------------------------------------------------------
+
+If extensions (or modules to document with autodoc) are in another directory,
+add these directories to sys.path here. If the directory is relative to the
+documentation root, use os.path.abspath to make it absolute, like shown here.
+"""
+
+import os
+import sys
+
+from sphinx.apidoc import main
+from mock import Mock as MagicMock
+
+sys.path.insert(0, os.path.abspath('.'))
+sys.path.insert(0, os.path.abspath('..'))
+sys.path.insert(0, os.path.abspath('../src'))
+sys.path.insert(0, os.path.abspath('../src/pyelliptic'))
+
+import version
+
+
+# -- Project information -----------------------------------------------------
+
+project = u'PyBitmessage'
+copyright = u'2018, The Bitmessage Team'  # pylint: disable=redefined-builtin
+author = u'The Bitmessage Team'
+
+# The short X.Y version
+version = unicode(version.softwareVersion)
+
+# The full version, including alpha/beta/rc tags
+release = version
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    # 'sphinx.ext.doctest',  # Currently disabled due to bad doctests
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.todo',
+    'sphinx.ext.coverage',
+    'sphinx.ext.imgmath',
+    'sphinx.ext.viewcode',
+    'm2r',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+source_suffix = ['.rst', '.md']
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path .
+exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+# Deal with long lines in source view
+html_theme_options = {
+    'page_width': '1366px',
+}
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'PyBitmessagedoc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'PyBitmessage.tex', u'PyBitmessage Documentation',
+     u'The Bitmessage Team', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'pybitmessage', u'PyBitmessage Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'PyBitmessage', u'PyBitmessage Documentation',
+     author, 'PyBitmessage', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+epub_author = author
+epub_publisher = author
+epub_copyright = copyright
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
+
+
+# -- Extension configuration -------------------------------------------------
+
+# -- Options for intersphinx extension ---------------------------------------
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {'https://docs.python.org/': None}
+
+# -- Options for todo extension ----------------------------------------------
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True

docs/contribute.dir/develop.dir/documentation.rst

@@ -0,0 +1,18 @@
+Documentation
+=============
+
+Sphinx is used to pull richly formatted comments out of code, merge them with hand-written documentation and render it
+in HTML and other formats.
+
+To build the docs, simply run `$ fab -H localhost build_docs` once you have set up Fabric.
+
+Restructured Text (RsT) vs MarkDown (md)
+----------------------------------------
+
+There's much on the internet about this. Suffice to say RsT_ is preferred for Python documentation while md is preferred for web markup or for certain other languages.
+
+.. _Rst: [http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html]` style is preferred, 
+
+`md` files can also be incorporated by using mdinclude directives in the indices. They are translated to RsT before rendering to the various formats. Headers are translated as a hard-coded level `(H1: =, H2: -, H3: ^, H4: ~, H5: ", H6: #`.  This represents a small consideration for both styles. If markup is not translated to rst well enough, switch to rst.
+
+

docs/contribute.dir/develop.dir/fabric.rst

@@ -0,0 +1,2 @@
+.. mdinclude:: fabfile/README.md
+

docs/contribute.dir/develop.dir/opsec.rst

@@ -0,0 +1,38 @@
+Operational Security
+====================
+
+Bitmessage has many features that are designed to protect your anonymity during normal use. There are other things that you must or should do if you value your anonymity.
+
+Castles in the sand
+-------------------
+
+You cannot build a strong castle on unstable foundations. If your operating system is not wholly owned by you then it is impossible to make guarantees about an application.
+
+ * Carefully choose your operating system
+ * Ensure your operating system is up to date
+ * Ensure any dependencies and requirements are up to date
+
+Extrordinary claims require extrordinary evidence
+-------------------------------------------------
+
+If we are to make bold claims about protecting your privacy we should demonstrate this by strong actions.
+
+- PGP signed commits
+- looking to audit
+- warrant canary
+
+Digital foootprint
+------------------
+
+Your internet use can reveal metadata you wouldn't expect. This can be connected with other information about you if you're not careful.
+
+ * Use separate addresses for different puprose
+ * Don't make the same mistakes all the time
+ * Your language use is unique. The more you type, the more you fingerprint yourself. The words you know and use often vs the words you don't know or use often.
+
+Cleaning history
+----------------
+
+ * Tell your browser not to store BitMessage addresses
+ * Microsoft Office seems to offer the ability to define sensitive informations types. If browsers don't already they may in the future. Consider adding `BM-\w{x..y}`.
+

docs/contribute.dir/develop.dir/overview.rst

@@ -0,0 +1,88 @@
+Developing 
+==========
+
+Devops tasks
+------------
+
+Bitmessage makes use of fabric_ to define tasks such as building documentation or running checks and tests on the code. If you can install 
+
+.. _fabric: https://fabfile.org
+
+Code style and linters
+----------------------
+
+We aim to be PEP8 compliant but we recognise that we have a long way still to go. Currently we have style and lint exceptions specified at the most specific place we can. We are ignoring certain issues project-wide in order to avoid alert-blindess, avoid style and lint regressions and to allow continuous integration to hook into the output from the tools. While it is hoped that all new changes pass the checks, fixing some existing violations are mini-projects in themselves. Current thinking on ignorable violations is reflected in the options and comments in setup.cfg. Module and line-level lint warnings represent refactoring opportunities.
+
+Pull requests
+-------------
+
+There is a template at PULL_REQUEST_TEMPLATE.md that appears in the pull-request description. Please replace this text with something appropriate to your changes based off the ideas in the template.
+
+Bike-shedding
+-------------
+
+Beyond having well-documented, Pythonic code with static analysis tool checks, extensive test coverage and powerful devops tools, what else can we have? Without violating any linters there is room for making arbirary decisions solely for the sake of project consistency. These are the stuff of the pedant's PR comments. Rather than have such conversations in PR comments, we can lay out the result of discussion here.
+
+I'm putting up a strawman for each topic here, mostly based on my memory of reading related Stack Overflow articles etc. If contributors feel strongly (and we don't have anything better to do) then maybe we can convince each other to update this section.
+
+Trailing vs hanging braces
+   Data
+      Hanging closing brace is preferred, trailing commas always to help reduce churn in diffs
+   Function, class, method args
+      Inline until line-length, then style as per data
+   Nesting
+      Functions
+         Short: group hanging close parentheses 
+         Long: one closing parentheses per line
+
+Single vs double quotes
+   Single quotes are preferred; less strain on the hands, eyes
+
+   Double quotes are only better so as to contain single quotes, but we want to contain doubles as often 
+
+Line continuation
+   Implicit parentheses continuation is preferred
+
+British vs American spelling
+   We should be consistent, it looks like we have American to British at approx 140 to 60 in the code. There's not enough occurrences that we couldn't be consistent one way or the other. It breaks my heart that British spelling could lose this one but I'm happy to 'z' things up for the sake of consistency. So I put forward British to be preferred. Either that strawman wins out, or I incite interest in ~bike-shedding~ guiding the direction of this crucial topic from others.
+
+Dependency graph
+----------------
+
+These images are not very useful right now but the aim is to tweak the settings of one or more of them to be informative, and/or divide them up into smaller grapghs.
+
+To re-build them, run `fab build_docs:dep_graphs=true`. Note that the dot graph takes a lot of time.
+
+.. figure:: ../../../../_static/deps-neato.png
+   :alt: neato graph of dependencies
+   :width: 100 pc
+
+   :index:`Neato` graph of dependencies
+
+.. figure:: ../../../../_static/deps-sfdp.png
+   :alt: SFDP graph of dependencies
+   :width: 100 pc
+   
+   :index:`SFDP` graph of dependencies
+
+.. figure:: ../../../../_static/deps-dot.png
+   :alt: Dot graph of dependencies
+   :width: 100 pc
+
+   :index:`Dot` graph of dependencies
+
+Key management
+--------------
+
+Nitro key
+^^^^^^^^^
+
+Regular contributors are enouraged to take further steps to protect their key and the Nitro Key (Start) is recommended by the BitMessage project for this purpose.
+
+Debian-quirks
+~~~~~~~~~~~~~
+
+Stretch makes use of the directory ~/.gnupg/private-keys-v1.d/ to store the private keys. This simplifies some steps of the Nitro Key instructions. See step 5 of Debian's subkeys_ wiki page
+
+.. _subkeys: https://wiki.debian.org/Subkeys
+

docs/contribute.dir/develop.dir/testing.rst

@@ -0,0 +1,4 @@
+Testing
+=======
+
+Currently there is a Travis job somewhere which runs python setup.py test. This doesn't find any tests when run locally for some reason.

docs/contribute.dir/develop.dir/todo.rst

@@ -0,0 +1,4 @@
+TODO list
+=========
+
+.. todolist::

docs/contribute.dir/develop.rst

@@ -0,0 +1,8 @@
+Developing
+==========
+
+.. toctree::
+   :maxdepth: 2
+   :glob:
+
+   develop.dir/*

docs/contribute.dir/processes.rst

@@ -0,0 +1,28 @@
+Processes
+=========
+
+In other to keep the Bitmessage project running the team run a number of systems and accounts that form the 
+development pipeline and continuous delivery process. We are always striving to improve the process. Towards
+that end it is documented here.
+
+
+Project website
+---------------
+
+The bitmessage website_
+
+Github
+------
+
+Our official Github_ account is Bitmessage. Our issue tracker is here as well.
+
+
+BitMessage
+----------
+
+We eat our own dog food! You can send us bug reports via the Bitmessage chan at xxx
+
+
+.. _website: https://bitmessage.org
+.. _Github: https://github.com/Bitmessage/PyBitmessage
+

docs/contribute.rst

@@ -0,0 +1,25 @@
+Contributing
+============
+
+.. toctree::
+   :maxdepth: 2
+   :glob:
+
+   contribute.dir/*
+
+
+- Report_
+- Develop_(develop)
+- Translate_
+- Donate_
+- Fork the code and open a PR_ on github
+- Search the `issue tracker` on github or open a  new issue
+- Send bug report to the chan
+
+.. _Report: https://github.com/Bitmessage/PyBitmessage/issues
+.. _Develop: https://github.com/Bitmessage/PyBitmessage
+.. _Translate: https://www.transifex.com/bitmessage-project/pybitmessage/
+.. _Donate: https://tip4commit.com/github/Bitmessage/PyBitmessage
+.. _PR: https://github.com/Bitmessage/PyBitmessage/pulls
+.. _`issue tracker`: https://github.com/Bitmessage/PyBitmessage/issues
+   contributing/*

docs/index.rst

@@ -0,0 +1,15 @@
+.. mdinclude:: ../README.md
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   usage
+   contribute
+   
+Indices and tables
+------------------
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/make.bat

@@ -0,0 +1,36 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+set SPHINXPROJ=PyBitmessage
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd

docs/usage.rst

@@ -0,0 +1,21 @@
+Usage
+=====
+
+GUI
+---
+
+Bitmessage has a PyQT GUI_
+
+CLI
+---
+
+Bitmessage has a CLI_
+
+API
+---
+
+Bitmessage has an XML-RPC API_
+
+.. _GUI: https://bitmessage.org/wiki/PyBitmessage_Help
+.. _CLI: https://bitmessage.org/wiki/PyBitmessage_Help
+.. _API: https://bitmessage.org/wiki/API_Reference

fabfile/README.md

@@ -85,3 +85,17 @@ Host github
     HostName github.com
     IdentityFile ~/.ssh/id_rsa_github
 ```
+
+# Ideas for further development
+
+## Smaller 
+
+ * Decorators and context managers are useful for accepting common params like verbosity, force or doing command-level help
+ * if `git status` or `git status --staged` produce results, prefer that to generate the file list
+
+
+## Larger
+
+ * Support documentation translations, aim for current transifex'ed languages
+ * Fabric 2 is finally out, go @bitprophet! Invoke/Fabric2 is a rewrite of Fabric supporting Python3. Probably makes
+ sense for us to stick to the battle-hardened 1.x branch, at least until we support Python3.

fabfile/__init__.py

@@ -17,7 +17,7 @@ For more help on a particular command
 
 from fabric.api import env
 
-from fabfile.tasks import code_quality
+from fabfile.tasks import code_quality, build_docs, push_docs, clean
 
 
 # Without this, `fab -l` would display the whole docstring as preamble
@@ -26,6 +26,9 @@ __doc__ = ""
 # This list defines which tasks are made available to the user
 __all__ = [
     "code_quality",
+    "build_docs",
+    "push_docs",
+    "clean",
 ]
 
 # Honour the user's ssh client configuration

fabfile/tasks.py

@@ -2,13 +2,16 @@
 """
 Fabric tasks for PyBitmessage devops operations.
 
-# pylint: disable=not-context-manager
+Note that where tasks declare params to be bools, they use coerce_bool() and so will accept any commandline (string)
+representation of true or false that coerce_bool() understands.
+
 """
 
 import os
 import sys
 
-from fabric.api import run, task, hide, cd
+from fabric.api import run, task, hide, cd, settings, sudo
+from fabric.contrib.project import rsync_project
 from fabvenv import virtualenv
 
 from fabfile.lib import (
@@ -16,6 +19,9 @@ from fabfile.lib import (
     get_filtered_pycodestyle_output, get_filtered_flake8_output, get_filtered_pylint_output,
 )
 
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(os.path.dirname(__file__)), 'src')))  # noqa:E402
+from version import softwareVersion  # pylint: disable=wrong-import-position
+
 
 def get_tool_results(file_list):
     """Take a list of files and resuln the results of applying the tools"""
@@ -116,6 +122,43 @@ def generate_file_list(filename):
     return file_list
 
 
+def create_dependency_graphs():
+    """
+    To better understand the relationship between methods, dependency graphs can be drawn between functions and
+    methods.
+
+    Since the resulting images are large and could be out of date on the next commit, storing them in the repo is
+    pointless. Instead, it makes sense to build a dependency graph for a particular, documented version of the code.
+
+    .. todo:: Consider saving a hash of the intermediate dotty file so unnecessary image generation can be avoided.
+
+    """
+    with virtualenv(VENV_ROOT):
+        with hide('running', 'stdout'):
+
+            # .. todo:: consider a better place to put this, use a particular commit
+            with cd(PROJECT_ROOT):
+                with settings(warn_only=True):
+                    if run('stat pyan').return_code:
+                        run('git clone https://github.com/davidfraser/pyan.git')
+            with cd(os.path.join(PROJECT_ROOT, 'pyan')):
+                run('git checkout pre-python3')
+
+            # .. todo:: Use better settings. This is MVP to make a diagram
+            with cd(PROJECT_ROOT):
+                file_list = run("find . -type f -name '*.py' ! -path './src/.eggs/*'").split('\r\n')
+                for cmd in [
+                        'neato -Goverlap=false -Tpng > deps-neato.png',
+                        'sfdp -Goverlap=false -Tpng > deps-sfdp.png',
+                        'dot -Goverlap=false -Tpng > deps-dot.png',
+                ]:
+                    pyan_cmd = './pyan/pyan.py {} --dot'.format(' '.join(file_list))
+                    sed_cmd = r"sed s'/http\-old/http_old/'g"  # dot doesn't like dashes
+                    run('|'.join([pyan_cmd, sed_cmd, cmd]))
+
+                run('mv *.png docs/_build/html/_static/')
+
+
 @task
 def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
     """
@@ -136,11 +179,9 @@ def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
     :type details: bool, default False
     :param fix: Run autopep8 aggressively on the displayed file(s)
     :type fix: bool, default False
-    :param filename: Rather than analysing all files and displaying / fixing the top N, just analyse / display / fix
-    the specified file
+    :param filename: Don't test/fix the top N, just the specified file
     :type filename: string, valid path to a file, default all files in the project
-    :return: This fabric task has an exit status equal to the total number of violations and uses stdio but it does
-    not return anything if you manage to call it successfully from Python
+    :return: None, exit status equals total number of violations
     :rtype: None
 
     Intended to be temporary until we have improved code quality and have safeguards to maintain it in place.
@@ -163,3 +204,93 @@ def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
 
     print_results(results, top, verbose, details)
     sys.exit(sum([item['total_violations'] for item in results]))
+
+
+@task
+def build_docs(dep_graph=False, apidoc=True):
+    """
+    Build the documentation locally.
+
+    :param dep_graph: Build the dependency graphs
+    :type dep_graph: Bool, default False
+    :param apidoc: Build the automatically generated rst files from the source code
+    :type apidoc: Bool, default True
+
+    Default usage:
+
+        $ fab -H localhost build_docs
+
+    Implementation:
+
+    First, a dependency graph is generated and converted into an image that is referenced in the development page.
+
+    Next, the sphinx-apidoc command is (usually) run which searches the code. As part of this it loads the modules and
+    if this has side-effects then they will be evident. Any documentation strings that make use of Python documentation
+    conventions (like parameter specification) or the Restructured Text (RsT) syntax will be extracted.
+
+    Next, the `make html` command is run to generate HTML output. Other formats (epub, pdf) are available.
+
+    .. todo:: support other languages
+
+    """
+
+    apidoc = coerce_bool(apidoc)
+
+    if coerce_bool(dep_graph):
+        create_dependency_graphs()
+
+    with virtualenv(VENV_ROOT):
+        with hide('running'):
+
+            apidoc_result = 0
+            if apidoc:
+                run('mkdir -p {}'.format(os.path.join(PROJECT_ROOT, 'docs', 'autodoc')))
+                with cd(os.path.join(PROJECT_ROOT, 'docs', 'autodoc')):
+                    with settings(warn_only=True):
+                        run('rm *.rst')
+                with cd(os.path.join(PROJECT_ROOT, 'docs')):
+                    apidoc_result = run('sphinx-apidoc -o autodoc ..').return_code
+
+            with cd(os.path.join(PROJECT_ROOT, 'docs')):
+                make_result = run('make html').return_code
+                return_code = apidoc_result + make_result
+
+    sys.exit(return_code)
+
+
+@task
+def push_docs(path=None):
+    """
+    Upload the generated docs to a public server.
+
+    Default usage:
+
+        $ fab -H localhost push_docs
+
+    .. todo:: support other languages
+    .. todo:: integrate with configuration management data to get web root and webserver restart command
+
+    """
+
+    # Making assumptions
+    WEB_ROOT = path if path is not None else os.path.join('var', 'www', 'html', 'pybitmessage', 'en', 'latest')
+    VERSION_ROOT = os.path.join(os.path.dirname(WEB_ROOT), softwareVersion)
+
+    rsync_project(
+        remote_dir=VERSION_ROOT,
+        local_dir=os.path.join(PROJECT_ROOT, 'docs', '_build', 'html')
+    )
+    result = run('ln -sf {0} {1}'.format(WEB_ROOT, VERSION_ROOT))
+    if result.return_code:
+        print 'Linking the new release failed'
+
+    # More assumptions
+    sudo('systemctl restart apache2')
+
+
+@task
+def clean():
+    """Clean up files generated by fabric commands."""
+    with hide('running', 'stdout'):
+        with cd(PROJECT_ROOT):
+            run(r"find . -name '*.pyc' -exec rm '{}' \;")

setup.py

@@ -15,7 +15,14 @@ EXTRAS_REQUIRE = {
     'pyopencl': ['pyopencl'],
     'prctl': ['python_prctl'],  # Named threads
     'qrcode': ['qrcode'],
-    'sound;platform_system=="Windows"': ['winsound']
+    'sound;platform_system=="Windows"': ['winsound'],
+    'docs': [
+        'sphinx',  # fab build_docs
+        'graphviz',  # fab build_docs
+        'curses',  # src/depends.py
+        'python2-pythondialog',  # src/depends.py
+        'm2r',  # fab build_docs
+    ],
 }
 
 

src/api_client.py

@@ -5,6 +5,8 @@ import xmlrpclib
 import json
 import time
 
+if __name__ == '__main__':
+
     api = xmlrpclib.ServerProxy("http://bradley:password@localhost:8442/")
 
     print 'Let\'s test the API first.'

src/debug.py

@@ -4,23 +4,26 @@ Logging and debuging facility
 =============================
 
 Levels:
-    DEBUG       Detailed information, typically of interest only when
-                diagnosing problems.
-    INFO        Confirmation that things are working as expected.
-    WARNING     An indication that something unexpected happened, or indicative
-                of some problem in the near future (e.g. ‘disk space low’).
-                The software is still working as expected.
-    ERROR       Due to a more serious problem, the software has not been able
-                to perform some function.
-    CRITICAL    A serious error, indicating that the program itself may be
-                unable to continue running.
+
+   DEBUG
+      Detailed information, typically of interest only when diagnosing problems.
+   INFO
+      Confirmation that things are working as expected.
+   WARNING
+      An indication that something unexpected happened, or indicative of some problem in the
+      near future (e.g. ‘disk space low’). The software is still working as expected.
+   ERROR
+      Due to a more serious problem, the software has not been able to perform some function.
+   CRITICAL
+      A serious error, indicating that the program itself may be unable to continue running.
 
 There are three loggers: `console_only`, `file_only` and `both`.
 
-Use: `from debug import logger` to import this facility into whatever module
-     you wish to log messages from. Logging is thread-safe so you don't have
-     to worry about locks, just import and log.
+Use: `from debug import logger` to import this facility into whatever module you wish to log messages from.
+    Logging is thread-safe so you don't have to worry about locks, just import and log.
+
 """
+
 import logging
 import logging.config
 import os
@@ -56,9 +59,8 @@ def configureLogging():
                 ' logging config\n%s' %
                 (os.path.join(state.appdata, 'logging.dat'), sys.exc_info()))
         else:
-            # no need to confuse the user if the logger config
-            # is missing entirely
-            print('Using default logger configuration')
+            # no need to confuse the user if the logger config is missing entirely
+            print "Using default logger configuration"
 
     sys.excepthook = log_uncaught_exceptions
 
@@ -111,6 +113,8 @@ def configureLogging():
     return True
 
 
+if __name__ == "__main__":
+
     # TODO (xj9): Get from a config file.
     #logger = logging.getLogger('console_only')
     if configureLogging():
@@ -120,7 +124,8 @@ if configureLogging():
             logger = logging.getLogger('both')
     else:
         logger = logging.getLogger('default')
-
+else:
+    logger = logging.getLogger('default')
 
 def restartLoggingInUpdatedAppdataLocation():
     global logger
@@ -135,3 +140,4 @@ def restartLoggingInUpdatedAppdataLocation():
             logger = logging.getLogger('both')
     else:
         logger = logging.getLogger('default')
+

src/depends.py

@@ -1,5 +1,6 @@
 #! python
 
+import subprocess
 import sys
 import os
 import pyelliptic.openssl
@@ -145,7 +146,10 @@ def check_openssl():
         except OSError:
             continue
         logger.info('OpenSSL Name: ' + library._name)
+        try:
             openssl_version, openssl_hexversion, openssl_cflags = pyelliptic.openssl.get_version(library)
+        except AttributeError:  # sphinx chokes
+            return True
         if not openssl_version:
             logger.error('Cannot determine version of this OpenSSL library.')
             return False
@@ -185,6 +189,12 @@ def check_curses():
     except ImportError:
         logger.error('The curses interface can not be used. The pythondialog package is not available.')
         return False
+    try:
+        subprocess.check_call('which dialog')
+    except subprocess.CalledProcessError:
+        logger.error('Curses requires the `dialog` command to be installed as well as the python library.')
+        return False
+
     logger.info('pythondialog Package Version: ' + dialog.__version__)
     dialog_util_version = dialog.Dialog().cached_backend_version
     #The pythondialog author does not like Python2 str, so we have to use

src/fallback/__init__.py

@@ -0,0 +1,3 @@
+"""
+.. todo:: hello world
+"""

src/fallback/umsgpack/umsgpack.py

@@ -205,7 +205,7 @@ load = None
 loads = None
 
 compatibility = False
-"""
+u"""
 Compatibility mode boolean.
 
 When compatibility mode is enabled, u-msgpack-python will serialize both

src/message_data_reader.py

@@ -1,14 +1,22 @@
-#This program can be used to print out everything in your Inbox or Sent folders and also take things out of the trash.
-#Scroll down to the bottom to see the functions that you can uncomment. Save then run this file.
-#The functions which only read the database file seem to function just fine even if you have Bitmessage running but you should definitly close it before running the functions that make changes (like taking items out of the trash).
+# pylint: disable=too-many-locals
+"""
+This program can be used to print out everything in your Inbox or Sent folders and also take things out of the trash.
+Scroll down to the bottom to see the functions that you can uncomment. Save then run this file.
+The functions which only read the database file seem to function just
+fine even if you have Bitmessage running but you should definitly close
+it before running the functions that make changes (like taking items out
+of the trash).
+"""
+
+from __future__ import absolute_import
 
 import sqlite3
+from binascii import hexlify
 from time import strftime, localtime
-import sys
+
 import paths
 import queues
-import state
-from binascii import hexlify
+
 
 appdata = paths.lookupAppdataFolder()
 
@@ -16,7 +24,9 @@ conn = sqlite3.connect( appdata + 'messages.dat' )
 conn.text_factory = str
 cur = conn.cursor()
 
+
 def readInbox():
+    """Print each row from inbox table"""
     print 'Printing everything in inbox table:'
     item = '''select * from inbox'''
     parameters = ''
@@ -25,17 +35,24 @@ def readInbox():
     for row in output:
         print row
 
+
 def readSent():
+    """Print each row from sent table"""
     print 'Printing everything in Sent table:'
     item = '''select * from sent where folder !='trash' '''
     parameters = ''
     cur.execute(item, parameters)
     output = cur.fetchall()
     for row in output:
-        msgid, toaddress, toripe, fromaddress, subject, message, ackdata, lastactiontime, sleeptill, status, retrynumber, folder, encodingtype, ttl = row
-        print hexlify(msgid), toaddress, 'toripe:', hexlify(toripe), 'fromaddress:', fromaddress, 'ENCODING TYPE:', encodingtype, 'SUBJECT:', repr(subject), 'MESSAGE:', repr(message), 'ACKDATA:', hexlify(ackdata), lastactiontime, status, retrynumber, folder
+        (msgid, toaddress, toripe, fromaddress, subject, message, ackdata, lastactiontime,
+         sleeptill, status, retrynumber, folder, encodingtype, ttl) = row  # pylint: disable=unused-variable
+        print(hexlify(msgid), toaddress, 'toripe:', hexlify(toripe), 'fromaddress:', fromaddress, 'ENCODING TYPE:',
+              encodingtype, 'SUBJECT:', repr(subject), 'MESSAGE:', repr(message), 'ACKDATA:', hexlify(ackdata),
+              lastactiontime, status, retrynumber, folder)
+
 
 def readSubscriptions():
+    """Print each row from subscriptions table"""
     print 'Printing everything in subscriptions table:'
     item = '''select * from subscriptions'''
     parameters = ''
@@ -44,7 +61,9 @@ def readSubscriptions():
     for row in output:
         print row
 
+
 def readPubkeys():
+    """Print each row from pubkeys table"""
     print 'Printing everything in pubkeys table:'
     item = '''select address, transmitdata, time, usedpersonally from pubkeys'''
     parameters = ''
@@ -52,61 +71,66 @@ def readPubkeys():
     output = cur.fetchall()
     for row in output:
         address, transmitdata, time, usedpersonally = row
-        print 'Address:', address, '\tTime first broadcast:', unicode(strftime('%a, %d %b %Y  %I:%M %p',localtime(time)),'utf-8'), '\tUsed by me personally:', usedpersonally, '\tFull pubkey message:', hexlify(transmitdata)
+        print(
+            'Address:', address, '\tTime first broadcast:', unicode(
+                strftime('%a, %d %b %Y  %I:%M %p', localtime(time)), 'utf-8'),
+            '\tUsed by me personally:', usedpersonally, '\tFull pubkey message:', hexlify(transmitdata),
+        )
+
 
 def readInventory():
+    """Print each row from inventory table"""
     print 'Printing everything in inventory table:'
     item = '''select hash, objecttype, streamnumber, payload, expirestime from inventory'''
     parameters = ''
     cur.execute(item, parameters)
     output = cur.fetchall()
     for row in output:
-        hash, objecttype, streamnumber, payload, expirestime = row
-        print 'Hash:', hexlify(hash), objecttype, streamnumber, '\t', hexlify(payload), '\t', unicode(strftime('%a, %d %b %Y  %I:%M %p',localtime(expirestime)),'utf-8')
+        obj_hash, objecttype, streamnumber, payload, expirestime = row
+        print 'Hash:', hexlify(obj_hash), objecttype, streamnumber, '\t', hexlify(payload), '\t', unicode(
+            strftime('%a, %d %b %Y  %I:%M %p', localtime(expirestime)), 'utf-8')
 
 
 def takeInboxMessagesOutOfTrash():
+    """Update all inbox messages with folder=trash to have folder=inbox"""
     item = '''update inbox set folder='inbox' where folder='trash' '''
     parameters = ''
     cur.execute(item, parameters)
-    output = cur.fetchall()
+    _ = cur.fetchall()
     conn.commit()
     print 'done'
 
+
 def takeSentMessagesOutOfTrash():
+    """Update all sent messages with folder=trash to have folder=sent"""
     item = '''update sent set folder='sent' where folder='trash' '''
     parameters = ''
     cur.execute(item, parameters)
-    output = cur.fetchall()
+    _ = cur.fetchall()
     conn.commit()
     print 'done'
 
+
 def markAllInboxMessagesAsUnread():
+    """Update all messages in inbox to have read=0"""
     item = '''update inbox set read='0' '''
     parameters = ''
     cur.execute(item, parameters)
-    output = cur.fetchall()
+    _ = cur.fetchall()
     conn.commit()
     queues.UISignalQueue.put(('changedInboxUnread', None))
     print 'done'
 
+
 def vacuum():
+    """Perform a vacuum on the database"""
     item = '''VACUUM'''
     parameters = ''
     cur.execute(item, parameters)
-    output = cur.fetchall()
+    _ = cur.fetchall()
     conn.commit()
     print 'done'
 
-#takeInboxMessagesOutOfTrash()
-#takeSentMessagesOutOfTrash()
-#markAllInboxMessagesAsUnread()
+
+if __name__ == '__main__':
     readInbox()
-#readSent()
-#readPubkeys()
-#readSubscriptions()
-#readInventory()
-#vacuum()  #will defragment and clean empty space from the messages.dat file.
-
-
-

src/pyelliptic/ecc.py

@@ -16,28 +16,28 @@ class ECC:
     Asymmetric encryption with Elliptic Curve Cryptography (ECC)
     ECDH, ECDSA and ECIES
 
-        import pyelliptic
+        >>> import pyelliptic
 
-        alice = pyelliptic.ECC() # default curve: sect283r1
-        bob = pyelliptic.ECC(curve='sect571r1')
+        >>> alice = pyelliptic.ECC() # default curve: sect283r1
+        >>> bob = pyelliptic.ECC(curve='sect571r1')
 
-        ciphertext = alice.encrypt("Hello Bob", bob.get_pubkey())
-        print bob.decrypt(ciphertext)
+        >>> ciphertext = alice.encrypt("Hello Bob", bob.get_pubkey())
+        >>> print bob.decrypt(ciphertext)
 
-        signature = bob.sign("Hello Alice")
-        # alice's job :
-        print pyelliptic.ECC(
-            pubkey=bob.get_pubkey()).verify(signature, "Hello Alice")
+        >>> signature = bob.sign("Hello Alice")
+        >>> # alice's job :
+        >>> print pyelliptic.ECC(
+        >>>     pubkey=bob.get_pubkey()).verify(signature, "Hello Alice")
 
-        # ERROR !!!
-        try:
-            key = alice.get_ecdh_key(bob.get_pubkey())
-        except: print("For ECDH key agreement,\
-                      the keys must be defined on the same curve !")
+        >>> # ERROR !!!
+        >>> try:
+        >>>     key = alice.get_ecdh_key(bob.get_pubkey())
+        >>> except:
+        >>>     print("For ECDH key agreement, the keys must be defined on the same curve !")
 
-        alice = pyelliptic.ECC(curve='sect571r1')
-        print alice.get_ecdh_key(bob.get_pubkey()).encode('hex')
-        print bob.get_ecdh_key(alice.get_pubkey()).encode('hex')
+        >>> alice = pyelliptic.ECC(curve='sect571r1')
+        >>> print alice.get_ecdh_key(bob.get_pubkey()).encode('hex')
+        >>> print bob.get_ecdh_key(alice.get_pubkey()).encode('hex')
 
     """
     def __init__(self, pubkey=None, privkey=None, pubkey_x=None,

src/socks/__init__.py

@@ -5,6 +5,7 @@ Copyright 2006 Dan-Haim. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification,
 are permitted provided that the following conditions are met:
+
  1. Redistributions of source code must retain the above copyright notice, this
     list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice,
@@ -146,19 +147,32 @@ class socksocket(socket.socket):
     def setproxy(self, proxytype=None, addr=None, port=None, rdns=True, username=None, password=None):
         """setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])
         Sets the proxy to be used.
-        proxytype -    The type of the proxy to be used. Three types
+
+         proxytype
+           The type of the proxy to be used. Three types
            are supported: PROXY_TYPE_SOCKS4 (including socks4a),
            PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP
-        addr -        The address of the server (IP or DNS).
-        port -        The port of the server. Defaults to 1080 for SOCKS
+
+        addr
+           The address of the server (IP or DNS).
+
+        port
+           The port of the server. Defaults to 1080 for SOCKS
            servers and 8080 for HTTP proxy servers.
-        rdns -        Should DNS queries be preformed on the remote side
+
+        rdns
+           Should DNS queries be preformed on the remote side
            (rather than the local side). The default is True.
            Note: This has no effect with SOCKS4 servers.
-        username -    Username to authenticate with to the server.
+
+        username
+           Username to authenticate with to the server.
            The default is no authentication.
-        password -    Password to authenticate with to the server.
+
+        password
+           Password to authenticate with to the server.
            Only relevant when username is also provided.
+
         """
         self.__proxy = (proxytype, addr, port, rdns, username, password)