Commit 3c0d2b1f authored by Cyril Roelandt's avatar Cyril Roelandt
Browse files

Initial commit.

parents
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
.static_storage/
.media/
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
# Changelog
All notable changes to this project will be documented in this file.
Based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
## [0.1] - 2018-02-17
### Added
- Initial CLI tool (upt)
- Initial library
- Support for user-defined frontends
- Support for user-defined backends
Copyright 2018 Cyril Roelandt
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# Universal Packaging Tool (upt)
A unified CLI tool that converts a package from a language-specific package
manager (such as PyPI or NPM) to an almost ready-to-use package for Free
Unix-based operating systems (such as a GNU/Linux distribution or \*BSD).
## Installing
Please note that Python 3.6 or newer is required to run upt.
You may install upt from PyPI:
$ pip install upt
You may also install it from the Git repository or a tarball:
$ python setup.py develop
By itself, upt is useless. It requires frontends and or backends to provide
useful features. See the list of available frontends/backends at the end of this
document. You may install them by running:
$ pip install upt-pypi
$ pip install upt-openbsd
$ pip install ...
## Usage
You may get help by running:
$ upt --help
Upt gathers data about a package using a **frontend** and writes a package
definition using a **backend**. You may list all installed frontends and
backends by running:
```
$ upt list-frontends
cpan
pypi
$ upt list-backends
guix
openbsd
```
To package [the requests library from
PyPI](https://pypi.python.org/pypi/requests) for [GNU
Guix](https://www.gnu.org/s/guix), just write:
$ upt package --frontend pypi --backend guix requests
Or, using shorter options:
$ upt package -f pypi -b guix requests
The package definition will be written to the standard output.
## Under the hood
There are two kinds of places from where packages may be installed:
- language specific package repositories (CPAN, CRAN, CTAN, Hackage, NPM, PyPI,
RubyGems)
- traditional, language agnostic package managers (apt for Debian, dnf for
Fedora, Nix, Guix, pkgsrc, the OpenBSD ports)
Packages available on the former are usually also packaged in the latter ones.
Since packaging is a tedious and repetitive task, packagers have written tools
to assist them in this endeavor: Guix has "guix import", a collection of scripts
that parse CPAN, PyPI and other language specific package repositories and
generate a Guix package definition; Debian has tools such as pypi2deb,
npm2deb... The same goes for RPM-based distributions.
Let's talk about PyPI. It seems that (almost) every single distribution has (or
will have) a tool that parses PyPI. Whatever the distribution may be (Debian,
Fedora, Guix...), parsing PyPI is basically always the same thing: it is just
reading a JSON document. This code is not shared among distributions though,
which means that they all end up writing the same kind of code. The same is true
for other language specific package managers.
The distribution specific code is usually not even shared among the various
tools provided by said distribution. Writing an OpenBSD port always involve
creating a directory, writing the maintainer's name to a Makefile, etc. These
operations are common to all packages: they should not be rewritten over and
over by every single tool OpenBSD developers may end up using. The same is true
for Debian: creating a Debian package involves a lot of operations that are not
truly specific to the piece of software being packaged.
We may also note that all these tools come with their own interface: packagers
working with more than one package repository and/or more than one distribution
therefore have to use different interfaces to perform similar tasks.
The Universal Packaging Tool (upt) aims at providing package maintainers with a
unified CLI that harnesses the power of a modular infrastructure, meant to
factorize as much code as possible. Upt is the "core" of the project. It exposes
a CLI and a small Python library that defines an internal representation for
packages. Two kind of objects have to talk to upt:
- frontends, which parse a package repository and return information about a
given package using upt's internal representation;
- backends, which turn the internal representation of a package into a valid
package definition for a language agnostic package manager.
This is basically what other common tools do: the [GCC
compiler](https://gcc.gnu.org/) has frontends for languages and backends for CPU
architectures. Similarly, [pandoc](https://pandoc.org/) can convert files from
one markup language to another using various frontends and backends.
```
| | | |
Package | upt | upt | upt | Package
Repositories | Frontends | | Backends | Definitions
| | | |
+------+ | +----------+ | +-----+ | +-------------+ | +----------+
| | | | | | | | | | | | | |
| PyPI |------|->| upt-pypi |-|->| |-|->| upt-guix |-|->| Guix pkg |
| | | | | | | | | | | | | |
+------+ | +----------+ | | | | +-------------+ | +----------+
| | | upt | | |
+------+ | +----------+ | | | | +-------------+ | +----------+
| | | | | | | | | | | | | |
| CPAN |------|->| upt-cpan |-|->| |-|->| upt-openbsd |-|->| Makefile |
| | | | | | | | | | | | | |
+------+ | +----------+ | +-----+ | +-------------+ | +----------+
| | | |
```
Here we can see that using the [upt-pypi](https://pypi.python.org/pypi/upt-pypi)
and [upt-cpan](https://pypi.python.org/pypi/upt-cpan) frontends, upt is able to
retrieve information about packages hosted on [CPAN](https://www.cpan.org/) and
[PyPI](https://pypi.python.org/). Then, using the
[upt-guix](https://pypi.python.org/pypi/upt-guix) backend, it can generate a
package definition suitable for [GNU Guix](https://www.gnu.org/s/guix); using
the [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) backend, it can
generate a Makefile suitable for [OpenBSD](https://www.openbsd.org)'s ports.
Using this modular approach, we solved our two issues:
- We will only ever need *one* PyPI parser, *one* CPAN parser, etc.
- We will only have one backend per distribution. Backends will not be truly
language-agnostic: the build systems will not be the same for Python packages
and Ruby packages, for instance. Still, a lot of code may be factorized: the
creation of the required directories, the writing of some common metadata...
Last but not least, frontends and backends are not part of the upt core project.
Every backend, every frontend is a separate project, living in its own
repository, written by its own team. This way, nobody has control over the full
project, and different communities may make different technical choices (testing
tools, coding standards, version control systems...) without having to impose
them upon everyone else.
## Contributing to upt
You may contribute by sending patches to tipecaml@gmail.com or by opening a pull
request.
Your patches should not break anything, so, prior to sending them, be sure to
run:
$ tox -eflake8
$ tox -epy36
You may also want to contribute to one of the frontends/backends: they are
separate projects, so you should not send your patches to upt.
## Adding frontends or backends
### Adding a frontend
Your setup.py should use the entry points like this:
```
setup(
...
entry_points = {
'upt.frontends': [
'myfrontend=upt_myfrontend.upt_myfrontend:MyFrontendParser',
]
},
...
)
```
Let us now look at your upt_myfrontend/upt_myfrontend.py file:
```
...
import upt
...
class MyFrontEndPackage(upt.Package):
pass
class MyFrontendParser(upt.Parser):
16 name = 'myfrontend'
def parse(self, pkg_name):
...
return MyFrontendPackage(pkg_name,
version=...,
homepage=...,
summary=...,
description=...,
download_urls=[url1, url2, ...],
requirements={
'build': [pkg1, ...],
'run': [pkg2, ...],
'test': [pkg3, ...]
})
```
The most interesting method here is MyFrontendParser.parse. It must return a upt.Package object (you may define a subclass such as MyFrontendPackage if you wish). Only the package name and its version must be passed to the constructor, all other fields are optional.
### Adding a backend
```
setup(
...
entry_points = {
'upt.backends': [
'mybackend=upt_mybackend.upt_mybackend:MyBackend',
]
},
...
)
```
## Known frontends and backends
The following frontends are supported:
- [upt-cpan](https://pypi.python.org/pypi/upt-cpan) a frontend for
[CPAN](https://www.cpan.org/)
- [upt-pypi](https://pypi.python.org/pypi/upt-pypi) a frontend for
[PyPI](https://pypi.python.org/)
The following backends are supported:
- [upt-guix](https://pypi.python.org/pypi/upt-guix) a backend for [GNU
Guix](https://www.gnu.org/s/guix)
- [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) a backend for
[OpenBSD](https://www.openbsd.org)
Please tell us about your own frontends/backends!
## License
This project is distributed under the [3-Clause BSD
License](https://opensource.org/licenses/BSD-3-Clause). See the LICENSE file.
# Copyright 2018 Cyril Roelandt
#
# Licensed under the 3-clause BSD license. See the LICENSE file.
from os import path
from setuptools import setup, find_packages
here = path.abspath(path.dirname(__file__))
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='upt',
author='Cyril Roelandt',
author_email='tipecaml@gmail.com',
url='https://framagit.org/upt/upt',
version='0.1.dev0',
description='Package software from any package manager to any distribution',
long_description=long_description,
classifiers=[
'Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3 :: Only',
],
install_requires = [],
packages = find_packages(),
entry_points = {
'console_scripts': ['upt = upt.upt:main'],
},
)
[tox]
envlist=flake8,py36
[testenv]
commands = python -m unittest
[flake8]
basepython = python3
deps = flake8
commands = flake8 --exclude=upt/__init__.py upt/
# Copyright 2018 Cyril Roelandt
#
# Licensed under the 3-clause BSD license. See the LICENSE file.
from .upt import Backend
from .upt import Frontend
from .upt import Package
from .upt import PackageRequirement
from .exceptions import InvalidPackageNameError
from .exceptions import UnhandledFrontendError
# Copyright 2018 Cyril Roelandt
#
# Licensed under the 3-clause BSD license. See the LICENSE file.
class InvalidPackageNameError(Exception):
"""Invalid package name or non-existing package."""
def __init__(self, backend, pkg_name):
self.backend = backend
self.pkg_name = pkg_name
def __str__(self):
return (f'The package {self.pkg_name} could not be found by backend '
f'{self.backend}')
class UnhandledFrontendError(Exception):
"""Frontend not supported by backend."""
def __init__(self, backend, frontend):
self.backend = backend
self.frontend = frontend
def __str__(self):
return (f'The {self.backend} backend does not work well with the '
f'{self.frontend} frontend (yet!). Unable to generate a '
f'package definition.')
# Copyright 2018 Cyril Roelandt
#
# Licensed under the 3-clause BSD license. See the LICENSE file.
import unittest
import upt
class TestPackageRequirement(unittest.TestCase):
def test_str_with_specifier(self):
pkg_req = upt.upt.PackageRequirement('foo', '>3.14')
self.assertEqual(str(pkg_req), 'foo (>3.14)')
def test_str_without_specifier(self):
pkg_req = upt.upt.PackageRequirement('foo')
self.assertEqual(str(pkg_req), 'foo')
class TestCommandLine(unittest.TestCase):
def setUp(self):
self.parser = upt.upt.create_parser()
def test_package_missing_frontend(self):
args = 'package -b guix requests'.split()
with self.assertRaises(SystemExit):
self.parser.parse_args(args)
def test_package_missing_backend(self):
args = 'package -f pypi requests'.split()
with self.assertRaises(SystemExit):
self.parser.parse_args(args)
def test_package_missing_frontend_and_backend(self):
args = 'package requests'.split()
with self.assertRaises(SystemExit):
self.parser.parse_args(args)
def test_package_missing_package(self):
args = 'package -f pypi -b guix'.split()
with self.assertRaises(SystemExit):
self.parser.parse_args(args)
# Copyright 2018 Cyril Roelandt
#
# Licensed under the 3-clause BSD license. See the LICENSE file.
import argparse
import sys
import pkg_resources
import upt.exceptions
class Backend(object):
pass
class Frontend(object):
"""Base class for all upt Frontends."""
pass
class PackageRequirement(object):
"""Package Requirement.
Two main attributes are used:
- name: a string, representing the name of the package.
- specifier: a string, representing a version specifier as defined in
https://www.python.org/dev/peps/pep-0440/#version-specifiers .
The format of the specifier is subject to changes.
"""
def __init__(self, name, specifier=None):
self.name = name
self.specifier = specifier
def __str__(self):
if self.specifier is not None:
return f'{self.name} ({self.specifier})'
else:
return f'{self.name}'
def __eq__(self, other):
return (self.name == other.name and
self.specifier == other.specifier)
class Package(object):
"""Base class for all packages.
Constructor arguments:
- name: required, the name of the package
- version: required, the version of the package, as a str
Optional keyword arguments:
- homepage: a string, representing the official homepage of the package
- summary: a short summary of the package's purpose
- description: a long description of the package's purpose
- download_urls: a list of strings, representing possible download URLs
- requirement: a dict with 3 keys:
- 'build': the build dependencies of the package;
- 'run': the runtime dependencies of the package;
- 'test': the test dependencies of the package.
The values associated with these keys are always lists of
PackageRequirement objects.
"""
def __init__(self, name, version, **kwargs):
self.name = name
self.version = version
self.homepage = kwargs.get('homepage', '')
self.summary = kwargs.get('summary', '')
self.description = kwargs.get('description', '')
self.download_urls = kwargs.get('download_urls', [])
# Software requirements, as instances of PackageRequirement: {
# 'build': [pkg1, ...],
# 'run': [pkg2, ...],
# 'test': [pkg3, ...]
# }
self.requirements = kwargs.get('requirements', {})
def __str__(self):
return f'{self.name}@{self.version}'
def create_parser():
parser = argparse.ArgumentParser(prog='upt')
subparsers = parser.add_subparsers(title='Commands', dest='cmd')
# List backends
subparsers.add_parser('list-backends',
help='List installed backends')
# List frontends
subparsers.add_parser('list-frontends',
help='List installed frontends')
# Actually package software
parser_package = subparsers.add_parser('package',
help='Package a piece of software')
required_args = parser_package.add_argument_group('Required arguments')
required_args.add_argument('-f', '--frontend', required=True,
help='Frontend to use')
required_args.add_argument('-b', '--backend', required=True,
help='Backend to use')
parser_package.add_argument('package', help='Name of the package')
return parser
def _get_installed_plugins(entrypoint):
plugins = {}
for ep in pkg_resources.iter_entry_points(entrypoint):
plugin_cls = ep.load()
plugins[plugin_cls.name] = plugin_cls
return plugins
def _get_installed_frontends():
return _get_installed_plugins('upt.frontends')
def _get_installed_backends():
return _get_installed_plugins('upt.backends')
def main():
parser = create_parser()
args = parser.parse_args()
if args.cmd == 'list-backends':
for backend in _get_installed_backends().keys():