Added: Sphinx docs, testing readthedocs integration

* Added: Sphinx configuration including readthedocs config
 * Added: Sphinx fabric task to auto-document the project
 * Fixed: Some issues in the code causing autodoc to fail when parsing
 * Added: Manual docs - structure, proof of concepts and RsT examples
 * Fixed: RsT formatting in docstrings
 * Fixed: Some adjacent minor style and lint issues
This commit is contained in:
coffeedogs 2018-05-26 13:43:21 +01:00
parent 1f6a7adf03
commit 3b75d900f6
No known key found for this signature in database
GPG Key ID: 9D818C503D0B7E70
30 changed files with 917 additions and 165 deletions

4
.gitignore vendored
View File

@ -3,6 +3,7 @@
**.DS_Store **.DS_Store
src/build src/build
src/dist src/dist
src/.eggs
src/.project src/.project
src/.pydevproject src/.pydevproject
src/.settings/ src/.settings/
@ -13,3 +14,6 @@ build/lib.*
build/temp.* build/temp.*
dist dist
*.egg-info *.egg-info
docs/_*/*
docs/autodoc/
pyan/

View File

@ -7,6 +7,5 @@ addons:
- build-essential - build-essential
- libcap-dev - libcap-dev
install: install:
- pip install -r requirements.txt
- python setup.py install - python setup.py install
script: pybitmessage -t script: pybitmessage -t

View File

@ -1,15 +1,33 @@
## Code contributions to the Bitmessage project ## Repository contributions to the PyBitmessage project
- try to explain what the code is about
- try to follow [PEP0008](https://www.python.org/dev/peps/pep-0008/)
- make the pull request against the ["v0.6" branch](https://github.com/Bitmessage/PyBitmessage/tree/v0.6)
- PGP-sign the commits included in the pull request
- try to use a good editor that removes trailing whitespace, highlights potential python issues and uses unix line endings
- You can get paid for merged commits if you register at [Tip4Commit](https://tip4commit.com/github/Bitmessage/PyBitmessage) - You can get paid for merged commits if you register at [Tip4Commit](https://tip4commit.com/github/Bitmessage/PyBitmessage)
If for some reason you don't want to use github, you can submit the patch using Bitmessage to the "bitmessage" chan, or to one of the developers. ### Code
- Try to refer to github issue tracker or other permanent sources of discussion about the issue.
- It is clear from the diff *what* you have done, it may be less clear *why* you have done it so explain why this change is necessary rather than what it does
### Documentation
- If there has been a change to the code, there's a good possibility there should be a corresponding change to the documentation
- If you can't run `fab build_docs` successfully, ask for someone to run it against your branch
### Tests
- If there has been a change to the code, there's a good possibility there should be a corresponding change to the tests
- If you can't run `fab tests` successfully, ask for someone to run it against your branch
## Translations ## Translations
For helping with translations, please use [Transifex](https://www.transifex.com/bitmessage-project/pybitmessage/). There is no need to submit pull requests for translations. - For helping with translations, please use [Transifex](https://www.transifex.com/bitmessage-project/pybitmessage/).
For translating technical terms it is recommended to consult the [Microsoft Language Portal](https://www.microsoft.com/Language/en-US/Default.aspx). - There is no need to submit pull requests for translations.
- For translating technical terms it is recommended to consult the [Microsoft Language Portal](https://www.microsoft.com/Language/en-US/Default.aspx).
### Gitiquette
- Make the pull request against the ["v0.6" branch](https://github.com/Bitmessage/PyBitmessage/tree/v0.6)
- PGP-sign the commits included in the pull request
- Use references to tickets, e.g. `addresses #123` or `fixes #234` in your commit messages
- Try to use a good editor that removes trailing whitespace, highlights potential python issues and uses unix line endings
- If for some reason you don't want to use github, you can submit the patch using Bitmessage to the "bitmessage" chan, or to one of the developers.

20
docs/Makefile Normal file
View File

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = PyBitmessage
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

210
docs/conf.py Normal file
View File

@ -0,0 +1,210 @@
# -*- coding: utf-8 -*-
"""
Configuration file for the Sphinx documentation builder.
This file does only contain a selection of the most common options. For a
full list see the documentation:
http://www.sphinx-doc.org/en/master/config
-- Path setup --------------------------------------------------------------
If extensions (or modules to document with autodoc) are in another directory,
add these directories to sys.path here. If the directory is relative to the
documentation root, use os.path.abspath to make it absolute, like shown here.
"""
import os
import sys
from sphinx.apidoc import main
from mock import Mock as MagicMock
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('..'))
sys.path.insert(0, os.path.abspath('../src'))
sys.path.insert(0, os.path.abspath('../src/pyelliptic'))
import version
# -- Project information -----------------------------------------------------
project = u'PyBitmessage'
copyright = u'2018, The Bitmessage Team' # pylint: disable=redefined-builtin
author = u'The Bitmessage Team'
# The short X.Y version
version = unicode(version.softwareVersion)
# The full version, including alpha/beta/rc tags
release = version
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
# 'sphinx.ext.doctest', # Currently disabled due to bad doctests
'sphinx.ext.intersphinx',
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.imgmath',
'sphinx.ext.viewcode',
'm2r',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_suffix = ['.rst', '.md']
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# Deal with long lines in source view
html_theme_options = {
'page_width': '1366px',
}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'PyBitmessagedoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'PyBitmessage.tex', u'PyBitmessage Documentation',
u'The Bitmessage Team', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'pybitmessage', u'PyBitmessage Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'PyBitmessage', u'PyBitmessage Documentation',
author, 'PyBitmessage', 'One line description of project.',
'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
epub_author = author
epub_publisher = author
epub_copyright = copyright
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# -- Extension configuration -------------------------------------------------
# -- Options for intersphinx extension ---------------------------------------
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True

View File

@ -0,0 +1,18 @@
Documentation
=============
Sphinx is used to pull richly formatted comments out of code, merge them with hand-written documentation and render it
in HTML and other formats.
To build the docs, simply run `$ fab -H localhost build_docs` once you have set up Fabric.
Restructured Text (RsT) vs MarkDown (md)
----------------------------------------
There's much on the internet about this. Suffice to say RsT_ is preferred for Python documentation while md is preferred for web markup or for certain other languages.
.. _Rst: [http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html]` style is preferred,
`md` files can also be incorporated by using mdinclude directives in the indices. They are translated to RsT before rendering to the various formats. Headers are translated as a hard-coded level `(H1: =, H2: -, H3: ^, H4: ~, H5: ", H6: #`. This represents a small consideration for both styles. If markup is not translated to rst well enough, switch to rst.

View File

@ -0,0 +1,2 @@
.. mdinclude:: fabfile/README.md

View File

@ -0,0 +1,38 @@
Operational Security
====================
Bitmessage has many features that are designed to protect your anonymity during normal use. There are other things that you must or should do if you value your anonymity.
Castles in the sand
-------------------
You cannot build a strong castle on unstable foundations. If your operating system is not wholly owned by you then it is impossible to make guarantees about an application.
* Carefully choose your operating system
* Ensure your operating system is up to date
* Ensure any dependencies and requirements are up to date
Extrordinary claims require extrordinary evidence
-------------------------------------------------
If we are to make bold claims about protecting your privacy we should demonstrate this by strong actions.
- PGP signed commits
- looking to audit
- warrant canary
Digital foootprint
------------------
Your internet use can reveal metadata you wouldn't expect. This can be connected with other information about you if you're not careful.
* Use separate addresses for different puprose
* Don't make the same mistakes all the time
* Your language use is unique. The more you type, the more you fingerprint yourself. The words you know and use often vs the words you don't know or use often.
Cleaning history
----------------
* Tell your browser not to store BitMessage addresses
* Microsoft Office seems to offer the ability to define sensitive informations types. If browsers don't already they may in the future. Consider adding `BM-\w{x..y}`.

View File

@ -0,0 +1,88 @@
Developing
==========
Devops tasks
------------
Bitmessage makes use of fabric_ to define tasks such as building documentation or running checks and tests on the code. If you can install
.. _fabric: https://fabfile.org
Code style and linters
----------------------
We aim to be PEP8 compliant but we recognise that we have a long way still to go. Currently we have style and lint exceptions specified at the most specific place we can. We are ignoring certain issues project-wide in order to avoid alert-blindess, avoid style and lint regressions and to allow continuous integration to hook into the output from the tools. While it is hoped that all new changes pass the checks, fixing some existing violations are mini-projects in themselves. Current thinking on ignorable violations is reflected in the options and comments in setup.cfg. Module and line-level lint warnings represent refactoring opportunities.
Pull requests
-------------
There is a template at PULL_REQUEST_TEMPLATE.md that appears in the pull-request description. Please replace this text with something appropriate to your changes based off the ideas in the template.
Bike-shedding
-------------
Beyond having well-documented, Pythonic code with static analysis tool checks, extensive test coverage and powerful devops tools, what else can we have? Without violating any linters there is room for making arbirary decisions solely for the sake of project consistency. These are the stuff of the pedant's PR comments. Rather than have such conversations in PR comments, we can lay out the result of discussion here.
I'm putting up a strawman for each topic here, mostly based on my memory of reading related Stack Overflow articles etc. If contributors feel strongly (and we don't have anything better to do) then maybe we can convince each other to update this section.
Trailing vs hanging braces
Data
Hanging closing brace is preferred, trailing commas always to help reduce churn in diffs
Function, class, method args
Inline until line-length, then style as per data
Nesting
Functions
Short: group hanging close parentheses
Long: one closing parentheses per line
Single vs double quotes
Single quotes are preferred; less strain on the hands, eyes
Double quotes are only better so as to contain single quotes, but we want to contain doubles as often
Line continuation
Implicit parentheses continuation is preferred
British vs American spelling
We should be consistent, it looks like we have American to British at approx 140 to 60 in the code. There's not enough occurrences that we couldn't be consistent one way or the other. It breaks my heart that British spelling could lose this one but I'm happy to 'z' things up for the sake of consistency. So I put forward British to be preferred. Either that strawman wins out, or I incite interest in ~bike-shedding~ guiding the direction of this crucial topic from others.
Dependency graph
----------------
These images are not very useful right now but the aim is to tweak the settings of one or more of them to be informative, and/or divide them up into smaller grapghs.
To re-build them, run `fab build_docs:dep_graphs=true`. Note that the dot graph takes a lot of time.
.. figure:: ../../../../_static/deps-neato.png
:alt: neato graph of dependencies
:width: 100 pc
:index:`Neato` graph of dependencies
.. figure:: ../../../../_static/deps-sfdp.png
:alt: SFDP graph of dependencies
:width: 100 pc
:index:`SFDP` graph of dependencies
.. figure:: ../../../../_static/deps-dot.png
:alt: Dot graph of dependencies
:width: 100 pc
:index:`Dot` graph of dependencies
Key management
--------------
Nitro key
^^^^^^^^^
Regular contributors are enouraged to take further steps to protect their key and the Nitro Key (Start) is recommended by the BitMessage project for this purpose.
Debian-quirks
~~~~~~~~~~~~~
Stretch makes use of the directory ~/.gnupg/private-keys-v1.d/ to store the private keys. This simplifies some steps of the Nitro Key instructions. See step 5 of Debian's subkeys_ wiki page
.. _subkeys: https://wiki.debian.org/Subkeys

View File

@ -0,0 +1,4 @@
Testing
=======
Currently there is a Travis job somewhere which runs python setup.py test. This doesn't find any tests when run locally for some reason.

View File

@ -0,0 +1,4 @@
TODO list
=========
.. todolist::

View File

@ -0,0 +1,8 @@
Developing
==========
.. toctree::
:maxdepth: 2
:glob:
develop.dir/*

View File

@ -0,0 +1,28 @@
Processes
=========
In other to keep the Bitmessage project running the team run a number of systems and accounts that form the
development pipeline and continuous delivery process. We are always striving to improve the process. Towards
that end it is documented here.
Project website
---------------
The bitmessage website_
Github
------
Our official Github_ account is Bitmessage. Our issue tracker is here as well.
BitMessage
----------
We eat our own dog food! You can send us bug reports via the Bitmessage chan at xxx
.. _website: https://bitmessage.org
.. _Github: https://github.com/Bitmessage/PyBitmessage

25
docs/contribute.rst Normal file
View File

@ -0,0 +1,25 @@
Contributing
============
.. toctree::
:maxdepth: 2
:glob:
contribute.dir/*
- Report_
- Develop_(develop)
- Translate_
- Donate_
- Fork the code and open a PR_ on github
- Search the `issue tracker` on github or open a new issue
- Send bug report to the chan
.. _Report: https://github.com/Bitmessage/PyBitmessage/issues
.. _Develop: https://github.com/Bitmessage/PyBitmessage
.. _Translate: https://www.transifex.com/bitmessage-project/pybitmessage/
.. _Donate: https://tip4commit.com/github/Bitmessage/PyBitmessage
.. _PR: https://github.com/Bitmessage/PyBitmessage/pulls
.. _`issue tracker`: https://github.com/Bitmessage/PyBitmessage/issues
contributing/*

15
docs/index.rst Normal file
View File

@ -0,0 +1,15 @@
.. mdinclude:: ../README.md
.. toctree::
:maxdepth: 2
overview
usage
contribute
Indices and tables
------------------
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

36
docs/make.bat Normal file
View File

@ -0,0 +1,36 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
set SPHINXPROJ=PyBitmessage
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd

21
docs/usage.rst Normal file
View File

@ -0,0 +1,21 @@
Usage
=====
GUI
---
Bitmessage has a PyQT GUI_
CLI
---
Bitmessage has a CLI_
API
---
Bitmessage has an XML-RPC API_
.. _GUI: https://bitmessage.org/wiki/PyBitmessage_Help
.. _CLI: https://bitmessage.org/wiki/PyBitmessage_Help
.. _API: https://bitmessage.org/wiki/API_Reference

View File

@ -85,3 +85,17 @@ Host github
HostName github.com HostName github.com
IdentityFile ~/.ssh/id_rsa_github IdentityFile ~/.ssh/id_rsa_github
``` ```
# Ideas for further development
## Smaller
* Decorators and context managers are useful for accepting common params like verbosity, force or doing command-level help
* if `git status` or `git status --staged` produce results, prefer that to generate the file list
## Larger
* Support documentation translations, aim for current transifex'ed languages
* Fabric 2 is finally out, go @bitprophet! Invoke/Fabric2 is a rewrite of Fabric supporting Python3. Probably makes
sense for us to stick to the battle-hardened 1.x branch, at least until we support Python3.

View File

@ -17,7 +17,7 @@ For more help on a particular command
from fabric.api import env from fabric.api import env
from fabfile.tasks import code_quality from fabfile.tasks import code_quality, build_docs, push_docs, clean
# Without this, `fab -l` would display the whole docstring as preamble # Without this, `fab -l` would display the whole docstring as preamble
@ -26,6 +26,9 @@ __doc__ = ""
# This list defines which tasks are made available to the user # This list defines which tasks are made available to the user
__all__ = [ __all__ = [
"code_quality", "code_quality",
"build_docs",
"push_docs",
"clean",
] ]
# Honour the user's ssh client configuration # Honour the user's ssh client configuration

View File

@ -2,13 +2,16 @@
""" """
Fabric tasks for PyBitmessage devops operations. Fabric tasks for PyBitmessage devops operations.
# pylint: disable=not-context-manager Note that where tasks declare params to be bools, they use coerce_bool() and so will accept any commandline (string)
representation of true or false that coerce_bool() understands.
""" """
import os import os
import sys import sys
from fabric.api import run, task, hide, cd from fabric.api import run, task, hide, cd, settings, sudo
from fabric.contrib.project import rsync_project
from fabvenv import virtualenv from fabvenv import virtualenv
from fabfile.lib import ( from fabfile.lib import (
@ -16,6 +19,9 @@ from fabfile.lib import (
get_filtered_pycodestyle_output, get_filtered_flake8_output, get_filtered_pylint_output, get_filtered_pycodestyle_output, get_filtered_flake8_output, get_filtered_pylint_output,
) )
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(os.path.dirname(__file__)), 'src'))) # noqa:E402
from version import softwareVersion # pylint: disable=wrong-import-position
def get_tool_results(file_list): def get_tool_results(file_list):
"""Take a list of files and resuln the results of applying the tools""" """Take a list of files and resuln the results of applying the tools"""
@ -116,6 +122,43 @@ def generate_file_list(filename):
return file_list return file_list
def create_dependency_graphs():
"""
To better understand the relationship between methods, dependency graphs can be drawn between functions and
methods.
Since the resulting images are large and could be out of date on the next commit, storing them in the repo is
pointless. Instead, it makes sense to build a dependency graph for a particular, documented version of the code.
.. todo:: Consider saving a hash of the intermediate dotty file so unnecessary image generation can be avoided.
"""
with virtualenv(VENV_ROOT):
with hide('running', 'stdout'):
# .. todo:: consider a better place to put this, use a particular commit
with cd(PROJECT_ROOT):
with settings(warn_only=True):
if run('stat pyan').return_code:
run('git clone https://github.com/davidfraser/pyan.git')
with cd(os.path.join(PROJECT_ROOT, 'pyan')):
run('git checkout pre-python3')
# .. todo:: Use better settings. This is MVP to make a diagram
with cd(PROJECT_ROOT):
file_list = run("find . -type f -name '*.py' ! -path './src/.eggs/*'").split('\r\n')
for cmd in [
'neato -Goverlap=false -Tpng > deps-neato.png',
'sfdp -Goverlap=false -Tpng > deps-sfdp.png',
'dot -Goverlap=false -Tpng > deps-dot.png',
]:
pyan_cmd = './pyan/pyan.py {} --dot'.format(' '.join(file_list))
sed_cmd = r"sed s'/http\-old/http_old/'g" # dot doesn't like dashes
run('|'.join([pyan_cmd, sed_cmd, cmd]))
run('mv *.png docs/_build/html/_static/')
@task @task
def code_quality(verbose=True, details=False, fix=False, filename=None, top=10): def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
""" """
@ -136,11 +179,9 @@ def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
:type details: bool, default False :type details: bool, default False
:param fix: Run autopep8 aggressively on the displayed file(s) :param fix: Run autopep8 aggressively on the displayed file(s)
:type fix: bool, default False :type fix: bool, default False
:param filename: Rather than analysing all files and displaying / fixing the top N, just analyse / display / fix :param filename: Don't test/fix the top N, just the specified file
the specified file
:type filename: string, valid path to a file, default all files in the project :type filename: string, valid path to a file, default all files in the project
:return: This fabric task has an exit status equal to the total number of violations and uses stdio but it does :return: None, exit status equals total number of violations
not return anything if you manage to call it successfully from Python
:rtype: None :rtype: None
Intended to be temporary until we have improved code quality and have safeguards to maintain it in place. Intended to be temporary until we have improved code quality and have safeguards to maintain it in place.
@ -163,3 +204,93 @@ def code_quality(verbose=True, details=False, fix=False, filename=None, top=10):
print_results(results, top, verbose, details) print_results(results, top, verbose, details)
sys.exit(sum([item['total_violations'] for item in results])) sys.exit(sum([item['total_violations'] for item in results]))
@task
def build_docs(dep_graph=False, apidoc=True):
"""
Build the documentation locally.
:param dep_graph: Build the dependency graphs
:type dep_graph: Bool, default False
:param apidoc: Build the automatically generated rst files from the source code
:type apidoc: Bool, default True
Default usage:
$ fab -H localhost build_docs
Implementation:
First, a dependency graph is generated and converted into an image that is referenced in the development page.
Next, the sphinx-apidoc command is (usually) run which searches the code. As part of this it loads the modules and
if this has side-effects then they will be evident. Any documentation strings that make use of Python documentation
conventions (like parameter specification) or the Restructured Text (RsT) syntax will be extracted.
Next, the `make html` command is run to generate HTML output. Other formats (epub, pdf) are available.
.. todo:: support other languages
"""
apidoc = coerce_bool(apidoc)
if coerce_bool(dep_graph):
create_dependency_graphs()
with virtualenv(VENV_ROOT):
with hide('running'):
apidoc_result = 0
if apidoc:
run('mkdir -p {}'.format(os.path.join(PROJECT_ROOT, 'docs', 'autodoc')))
with cd(os.path.join(PROJECT_ROOT, 'docs', 'autodoc')):
with settings(warn_only=True):
run('rm *.rst')
with cd(os.path.join(PROJECT_ROOT, 'docs')):
apidoc_result = run('sphinx-apidoc -o autodoc ..').return_code
with cd(os.path.join(PROJECT_ROOT, 'docs')):
make_result = run('make html').return_code
return_code = apidoc_result + make_result
sys.exit(return_code)
@task
def push_docs(path=None):
"""
Upload the generated docs to a public server.
Default usage:
$ fab -H localhost push_docs
.. todo:: support other languages
.. todo:: integrate with configuration management data to get web root and webserver restart command
"""
# Making assumptions
WEB_ROOT = path if path is not None else os.path.join('var', 'www', 'html', 'pybitmessage', 'en', 'latest')
VERSION_ROOT = os.path.join(os.path.dirname(WEB_ROOT), softwareVersion)
rsync_project(
remote_dir=VERSION_ROOT,
local_dir=os.path.join(PROJECT_ROOT, 'docs', '_build', 'html')
)
result = run('ln -sf {0} {1}'.format(WEB_ROOT, VERSION_ROOT))
if result.return_code:
print 'Linking the new release failed'
# More assumptions
sudo('systemctl restart apache2')
@task
def clean():
"""Clean up files generated by fabric commands."""
with hide('running', 'stdout'):
with cd(PROJECT_ROOT):
run(r"find . -name '*.pyc' -exec rm '{}' \;")

View File

View File

@ -15,7 +15,14 @@ EXTRAS_REQUIRE = {
'pyopencl': ['pyopencl'], 'pyopencl': ['pyopencl'],
'prctl': ['python_prctl'], # Named threads 'prctl': ['python_prctl'], # Named threads
'qrcode': ['qrcode'], 'qrcode': ['qrcode'],
'sound;platform_system=="Windows"': ['winsound'] 'sound;platform_system=="Windows"': ['winsound'],
'docs': [
'sphinx', # fab build_docs
'graphviz', # fab build_docs
'curses', # src/depends.py
'python2-pythondialog', # src/depends.py
'm2r', # fab build_docs
],
} }

View File

@ -5,6 +5,8 @@ import xmlrpclib
import json import json
import time import time
if __name__ == '__main__':
api = xmlrpclib.ServerProxy("http://bradley:password@localhost:8442/") api = xmlrpclib.ServerProxy("http://bradley:password@localhost:8442/")
print 'Let\'s test the API first.' print 'Let\'s test the API first.'

View File

@ -4,23 +4,26 @@ Logging and debuging facility
============================= =============================
Levels: Levels:
DEBUG Detailed information, typically of interest only when
diagnosing problems. DEBUG
INFO Confirmation that things are working as expected. Detailed information, typically of interest only when diagnosing problems.
WARNING An indication that something unexpected happened, or indicative INFO
of some problem in the near future (e.g. disk space low). Confirmation that things are working as expected.
The software is still working as expected. WARNING
ERROR Due to a more serious problem, the software has not been able An indication that something unexpected happened, or indicative of some problem in the
to perform some function. near future (e.g. disk space low). The software is still working as expected.
CRITICAL A serious error, indicating that the program itself may be ERROR
unable to continue running. Due to a more serious problem, the software has not been able to perform some function.
CRITICAL
A serious error, indicating that the program itself may be unable to continue running.
There are three loggers: `console_only`, `file_only` and `both`. There are three loggers: `console_only`, `file_only` and `both`.
Use: `from debug import logger` to import this facility into whatever module Use: `from debug import logger` to import this facility into whatever module you wish to log messages from.
you wish to log messages from. Logging is thread-safe so you don't have Logging is thread-safe so you don't have to worry about locks, just import and log.
to worry about locks, just import and log.
""" """
import logging import logging
import logging.config import logging.config
import os import os
@ -56,9 +59,8 @@ def configureLogging():
' logging config\n%s' % ' logging config\n%s' %
(os.path.join(state.appdata, 'logging.dat'), sys.exc_info())) (os.path.join(state.appdata, 'logging.dat'), sys.exc_info()))
else: else:
# no need to confuse the user if the logger config # no need to confuse the user if the logger config is missing entirely
# is missing entirely print "Using default logger configuration"
print('Using default logger configuration')
sys.excepthook = log_uncaught_exceptions sys.excepthook = log_uncaught_exceptions
@ -111,6 +113,8 @@ def configureLogging():
return True return True
if __name__ == "__main__":
# TODO (xj9): Get from a config file. # TODO (xj9): Get from a config file.
#logger = logging.getLogger('console_only') #logger = logging.getLogger('console_only')
if configureLogging(): if configureLogging():
@ -120,7 +124,8 @@ if configureLogging():
logger = logging.getLogger('both') logger = logging.getLogger('both')
else: else:
logger = logging.getLogger('default') logger = logging.getLogger('default')
else:
logger = logging.getLogger('default')
def restartLoggingInUpdatedAppdataLocation(): def restartLoggingInUpdatedAppdataLocation():
global logger global logger
@ -135,3 +140,4 @@ def restartLoggingInUpdatedAppdataLocation():
logger = logging.getLogger('both') logger = logging.getLogger('both')
else: else:
logger = logging.getLogger('default') logger = logging.getLogger('default')

View File

@ -1,5 +1,6 @@
#! python #! python
import subprocess
import sys import sys
import os import os
import pyelliptic.openssl import pyelliptic.openssl
@ -145,7 +146,10 @@ def check_openssl():
except OSError: except OSError:
continue continue
logger.info('OpenSSL Name: ' + library._name) logger.info('OpenSSL Name: ' + library._name)
try:
openssl_version, openssl_hexversion, openssl_cflags = pyelliptic.openssl.get_version(library) openssl_version, openssl_hexversion, openssl_cflags = pyelliptic.openssl.get_version(library)
except AttributeError: # sphinx chokes
return True
if not openssl_version: if not openssl_version:
logger.error('Cannot determine version of this OpenSSL library.') logger.error('Cannot determine version of this OpenSSL library.')
return False return False
@ -185,6 +189,12 @@ def check_curses():
except ImportError: except ImportError:
logger.error('The curses interface can not be used. The pythondialog package is not available.') logger.error('The curses interface can not be used. The pythondialog package is not available.')
return False return False
try:
subprocess.check_call('which dialog')
except subprocess.CalledProcessError:
logger.error('Curses requires the `dialog` command to be installed as well as the python library.')
return False
logger.info('pythondialog Package Version: ' + dialog.__version__) logger.info('pythondialog Package Version: ' + dialog.__version__)
dialog_util_version = dialog.Dialog().cached_backend_version dialog_util_version = dialog.Dialog().cached_backend_version
#The pythondialog author does not like Python2 str, so we have to use #The pythondialog author does not like Python2 str, so we have to use

View File

@ -0,0 +1,3 @@
"""
.. todo:: hello world
"""

View File

@ -205,7 +205,7 @@ load = None
loads = None loads = None
compatibility = False compatibility = False
""" u"""
Compatibility mode boolean. Compatibility mode boolean.
When compatibility mode is enabled, u-msgpack-python will serialize both When compatibility mode is enabled, u-msgpack-python will serialize both

View File

@ -1,14 +1,22 @@
#This program can be used to print out everything in your Inbox or Sent folders and also take things out of the trash. # pylint: disable=too-many-locals
#Scroll down to the bottom to see the functions that you can uncomment. Save then run this file. """
#The functions which only read the database file seem to function just fine even if you have Bitmessage running but you should definitly close it before running the functions that make changes (like taking items out of the trash). This program can be used to print out everything in your Inbox or Sent folders and also take things out of the trash.
Scroll down to the bottom to see the functions that you can uncomment. Save then run this file.
The functions which only read the database file seem to function just
fine even if you have Bitmessage running but you should definitly close
it before running the functions that make changes (like taking items out
of the trash).
"""
from __future__ import absolute_import
import sqlite3 import sqlite3
from binascii import hexlify
from time import strftime, localtime from time import strftime, localtime
import sys
import paths import paths
import queues import queues
import state
from binascii import hexlify
appdata = paths.lookupAppdataFolder() appdata = paths.lookupAppdataFolder()
@ -16,7 +24,9 @@ conn = sqlite3.connect( appdata + 'messages.dat' )
conn.text_factory = str conn.text_factory = str
cur = conn.cursor() cur = conn.cursor()
def readInbox(): def readInbox():
"""Print each row from inbox table"""
print 'Printing everything in inbox table:' print 'Printing everything in inbox table:'
item = '''select * from inbox''' item = '''select * from inbox'''
parameters = '' parameters = ''
@ -25,17 +35,24 @@ def readInbox():
for row in output: for row in output:
print row print row
def readSent(): def readSent():
"""Print each row from sent table"""
print 'Printing everything in Sent table:' print 'Printing everything in Sent table:'
item = '''select * from sent where folder !='trash' ''' item = '''select * from sent where folder !='trash' '''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() output = cur.fetchall()
for row in output: for row in output:
msgid, toaddress, toripe, fromaddress, subject, message, ackdata, lastactiontime, sleeptill, status, retrynumber, folder, encodingtype, ttl = row (msgid, toaddress, toripe, fromaddress, subject, message, ackdata, lastactiontime,
print hexlify(msgid), toaddress, 'toripe:', hexlify(toripe), 'fromaddress:', fromaddress, 'ENCODING TYPE:', encodingtype, 'SUBJECT:', repr(subject), 'MESSAGE:', repr(message), 'ACKDATA:', hexlify(ackdata), lastactiontime, status, retrynumber, folder sleeptill, status, retrynumber, folder, encodingtype, ttl) = row # pylint: disable=unused-variable
print(hexlify(msgid), toaddress, 'toripe:', hexlify(toripe), 'fromaddress:', fromaddress, 'ENCODING TYPE:',
encodingtype, 'SUBJECT:', repr(subject), 'MESSAGE:', repr(message), 'ACKDATA:', hexlify(ackdata),
lastactiontime, status, retrynumber, folder)
def readSubscriptions(): def readSubscriptions():
"""Print each row from subscriptions table"""
print 'Printing everything in subscriptions table:' print 'Printing everything in subscriptions table:'
item = '''select * from subscriptions''' item = '''select * from subscriptions'''
parameters = '' parameters = ''
@ -44,7 +61,9 @@ def readSubscriptions():
for row in output: for row in output:
print row print row
def readPubkeys(): def readPubkeys():
"""Print each row from pubkeys table"""
print 'Printing everything in pubkeys table:' print 'Printing everything in pubkeys table:'
item = '''select address, transmitdata, time, usedpersonally from pubkeys''' item = '''select address, transmitdata, time, usedpersonally from pubkeys'''
parameters = '' parameters = ''
@ -52,61 +71,66 @@ def readPubkeys():
output = cur.fetchall() output = cur.fetchall()
for row in output: for row in output:
address, transmitdata, time, usedpersonally = row address, transmitdata, time, usedpersonally = row
print 'Address:', address, '\tTime first broadcast:', unicode(strftime('%a, %d %b %Y %I:%M %p',localtime(time)),'utf-8'), '\tUsed by me personally:', usedpersonally, '\tFull pubkey message:', hexlify(transmitdata) print(
'Address:', address, '\tTime first broadcast:', unicode(
strftime('%a, %d %b %Y %I:%M %p', localtime(time)), 'utf-8'),
'\tUsed by me personally:', usedpersonally, '\tFull pubkey message:', hexlify(transmitdata),
)
def readInventory(): def readInventory():
"""Print each row from inventory table"""
print 'Printing everything in inventory table:' print 'Printing everything in inventory table:'
item = '''select hash, objecttype, streamnumber, payload, expirestime from inventory''' item = '''select hash, objecttype, streamnumber, payload, expirestime from inventory'''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() output = cur.fetchall()
for row in output: for row in output:
hash, objecttype, streamnumber, payload, expirestime = row obj_hash, objecttype, streamnumber, payload, expirestime = row
print 'Hash:', hexlify(hash), objecttype, streamnumber, '\t', hexlify(payload), '\t', unicode(strftime('%a, %d %b %Y %I:%M %p',localtime(expirestime)),'utf-8') print 'Hash:', hexlify(obj_hash), objecttype, streamnumber, '\t', hexlify(payload), '\t', unicode(
strftime('%a, %d %b %Y %I:%M %p', localtime(expirestime)), 'utf-8')
def takeInboxMessagesOutOfTrash(): def takeInboxMessagesOutOfTrash():
"""Update all inbox messages with folder=trash to have folder=inbox"""
item = '''update inbox set folder='inbox' where folder='trash' ''' item = '''update inbox set folder='inbox' where folder='trash' '''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() _ = cur.fetchall()
conn.commit() conn.commit()
print 'done' print 'done'
def takeSentMessagesOutOfTrash(): def takeSentMessagesOutOfTrash():
"""Update all sent messages with folder=trash to have folder=sent"""
item = '''update sent set folder='sent' where folder='trash' ''' item = '''update sent set folder='sent' where folder='trash' '''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() _ = cur.fetchall()
conn.commit() conn.commit()
print 'done' print 'done'
def markAllInboxMessagesAsUnread(): def markAllInboxMessagesAsUnread():
"""Update all messages in inbox to have read=0"""
item = '''update inbox set read='0' ''' item = '''update inbox set read='0' '''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() _ = cur.fetchall()
conn.commit() conn.commit()
queues.UISignalQueue.put(('changedInboxUnread', None)) queues.UISignalQueue.put(('changedInboxUnread', None))
print 'done' print 'done'
def vacuum(): def vacuum():
"""Perform a vacuum on the database"""
item = '''VACUUM''' item = '''VACUUM'''
parameters = '' parameters = ''
cur.execute(item, parameters) cur.execute(item, parameters)
output = cur.fetchall() _ = cur.fetchall()
conn.commit() conn.commit()
print 'done' print 'done'
#takeInboxMessagesOutOfTrash()
#takeSentMessagesOutOfTrash() if __name__ == '__main__':
#markAllInboxMessagesAsUnread()
readInbox() readInbox()
#readSent()
#readPubkeys()
#readSubscriptions()
#readInventory()
#vacuum() #will defragment and clean empty space from the messages.dat file.

View File

@ -16,28 +16,28 @@ class ECC:
Asymmetric encryption with Elliptic Curve Cryptography (ECC) Asymmetric encryption with Elliptic Curve Cryptography (ECC)
ECDH, ECDSA and ECIES ECDH, ECDSA and ECIES
import pyelliptic >>> import pyelliptic
alice = pyelliptic.ECC() # default curve: sect283r1 >>> alice = pyelliptic.ECC() # default curve: sect283r1
bob = pyelliptic.ECC(curve='sect571r1') >>> bob = pyelliptic.ECC(curve='sect571r1')
ciphertext = alice.encrypt("Hello Bob", bob.get_pubkey()) >>> ciphertext = alice.encrypt("Hello Bob", bob.get_pubkey())
print bob.decrypt(ciphertext) >>> print bob.decrypt(ciphertext)
signature = bob.sign("Hello Alice") >>> signature = bob.sign("Hello Alice")
# alice's job : >>> # alice's job :
print pyelliptic.ECC( >>> print pyelliptic.ECC(
pubkey=bob.get_pubkey()).verify(signature, "Hello Alice") >>> pubkey=bob.get_pubkey()).verify(signature, "Hello Alice")
# ERROR !!! >>> # ERROR !!!
try: >>> try:
key = alice.get_ecdh_key(bob.get_pubkey()) >>> key = alice.get_ecdh_key(bob.get_pubkey())
except: print("For ECDH key agreement,\ >>> except:
the keys must be defined on the same curve !") >>> print("For ECDH key agreement, the keys must be defined on the same curve !")
alice = pyelliptic.ECC(curve='sect571r1') >>> alice = pyelliptic.ECC(curve='sect571r1')
print alice.get_ecdh_key(bob.get_pubkey()).encode('hex') >>> print alice.get_ecdh_key(bob.get_pubkey()).encode('hex')
print bob.get_ecdh_key(alice.get_pubkey()).encode('hex') >>> print bob.get_ecdh_key(alice.get_pubkey()).encode('hex')
""" """
def __init__(self, pubkey=None, privkey=None, pubkey_x=None, def __init__(self, pubkey=None, privkey=None, pubkey_x=None,

View File

@ -5,6 +5,7 @@ Copyright 2006 Dan-Haim. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met: are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this 1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer. list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, 2. Redistributions in binary form must reproduce the above copyright notice,
@ -146,19 +147,32 @@ class socksocket(socket.socket):
def setproxy(self, proxytype=None, addr=None, port=None, rdns=True, username=None, password=None): def setproxy(self, proxytype=None, addr=None, port=None, rdns=True, username=None, password=None):
"""setproxy(proxytype, addr[, port[, rdns[, username[, password]]]]) """setproxy(proxytype, addr[, port[, rdns[, username[, password]]]])
Sets the proxy to be used. Sets the proxy to be used.
proxytype - The type of the proxy to be used. Three types
proxytype
The type of the proxy to be used. Three types
are supported: PROXY_TYPE_SOCKS4 (including socks4a), are supported: PROXY_TYPE_SOCKS4 (including socks4a),
PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP
addr - The address of the server (IP or DNS).
port - The port of the server. Defaults to 1080 for SOCKS addr
The address of the server (IP or DNS).
port
The port of the server. Defaults to 1080 for SOCKS
servers and 8080 for HTTP proxy servers. servers and 8080 for HTTP proxy servers.
rdns - Should DNS queries be preformed on the remote side
rdns
Should DNS queries be preformed on the remote side
(rather than the local side). The default is True. (rather than the local side). The default is True.
Note: This has no effect with SOCKS4 servers. Note: This has no effect with SOCKS4 servers.
username - Username to authenticate with to the server.
username
Username to authenticate with to the server.
The default is no authentication. The default is no authentication.
password - Password to authenticate with to the server.
password
Password to authenticate with to the server.
Only relevant when username is also provided. Only relevant when username is also provided.
""" """
self.__proxy = (proxytype, addr, port, rdns, username, password) self.__proxy = (proxytype, addr, port, rdns, username, password)