README.md 15.3 KB
Newer Older
Cyril Roelandt's avatar
Cyril Roelandt committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Universal Packaging Tool (upt)

A unified CLI tool that converts a package from a language-specific package
manager (such as PyPI or NPM) to an almost ready-to-use package for Free
Unix-based operating systems (such as a GNU/Linux distribution or \*BSD).


## Installing
Please note that Python 3.6 or newer is required to run upt.

You may install upt from PyPI:

    $ pip install upt

You may also install it from the Git repository or a tarball:

17
    $ python setup.py install
Cyril Roelandt's avatar
Cyril Roelandt committed
18
19
20
21
22
23
24
25
26

By itself, upt is useless. It requires frontends and or backends to provide
useful features. See the list of available frontends/backends at the end of this
document. You may install them by running:

    $ pip install upt-pypi
    $ pip install upt-openbsd
    $ pip install ...

27
28
29
30
31
32
33
34
35
36
37
38
You may also install upt and all its frontends by running:

    $ pip install upt[frontends]

You may also install upt and all its backends by running:

    $ pip install upt[backends]

You may also install upt and all of its frontends and backends by running:

    $ pip install upt[frontends,backends]

Cyril Roelandt's avatar
Cyril Roelandt committed
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

## Usage
You may get help by running:

    $ upt --help

Upt gathers data about a package using a **frontend** and writes a package
definition using a **backend**. You may list all installed frontends and
backends by running:

```
$ upt list-frontends
cpan
pypi
$ upt list-backends
guix
openbsd
```

To package [the requests library from
PyPI](https://pypi.python.org/pypi/requests) for [GNU
Guix](https://www.gnu.org/s/guix), just write:

    $ upt package --frontend pypi --backend guix requests

Or, using shorter options:

    $ upt package -f pypi -b guix requests

The package definition will be written to the standard output.


## Under the hood
There are two kinds of places from where packages may be installed:

- language specific package repositories (CPAN, CRAN, CTAN, Hackage, NPM, PyPI,
75
  RubyGems, etc.)
Cyril Roelandt's avatar
Cyril Roelandt committed
76
- traditional, language agnostic package managers (apt for Debian, dnf for
77
  Fedora, Nix, Guix, pkgsrc, the OpenBSD ports, etc.)
Cyril Roelandt's avatar
Cyril Roelandt committed
78
79
80
81
82
83

Packages available on the former are usually also packaged in the latter ones.
Since packaging is a tedious and repetitive task, packagers have written tools
to assist them in this endeavor: Guix has "guix import", a collection of scripts
that parse CPAN, PyPI and other language specific package repositories and
generate a Guix package definition; Debian has tools such as pypi2deb,
84
85
npm2deb... The same goes for RPM-based distributions, and other operating
systems.
Cyril Roelandt's avatar
Cyril Roelandt committed
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

Let's talk about PyPI. It seems that (almost) every single distribution has (or
will have) a tool that parses PyPI. Whatever the distribution may be (Debian,
Fedora, Guix...), parsing PyPI is basically always the same thing: it is just
reading a JSON document. This code is not shared among distributions though,
which means that they all end up writing the same kind of code. The same is true
for other language specific package managers.

The distribution specific code is usually not even shared among the various
tools provided by said distribution. Writing an OpenBSD port always involve
creating a directory, writing the maintainer's name to a Makefile, etc. These
operations are common to all packages: they should not be rewritten over and
over by every single tool OpenBSD developers may end up using. The same is true
for Debian: creating a Debian package involves a lot of operations that are not
truly specific to the piece of software being packaged.

We may also note that all these tools come with their own interface: packagers
working with more than one package repository and/or more than one distribution
therefore have to use different interfaces to perform similar tasks.

The Universal Packaging Tool (upt) aims at providing package maintainers with a
unified CLI that harnesses the power of a modular infrastructure, meant to
factorize as much code as possible. Upt is the "core" of the project. It exposes
a CLI and a small Python library that defines an internal representation for
packages. Two kind of objects have to talk to upt:

- frontends, which parse a package repository and return information about a
  given package using upt's internal representation;
- backends, which turn the internal representation of a package into a valid
  package definition for a language agnostic package manager.

This is basically what other common tools do: the [GCC
compiler](https://gcc.gnu.org/) has frontends for languages and backends for CPU
architectures. Similarly, [pandoc](https://pandoc.org/) can convert files from
one markup language to another using various frontends and backends.

```
                |               |          |                  |
   Package      |      upt      |    upt   |       upt        |    Package
 Repositories   |   Frontends   |          |     Backends     |  Definitions
                |               |          |                  |
  +------+      |  +----------+ |  +-----+ |  +-------------+ |  +----------+
  |      |      |  |          | |  |     | |  |             | |  |          |
  | PyPI |------|->| upt-pypi |-|->|     |-|->|  upt-guix   |-|->| Guix pkg |
  |      |      |  |          | |  |     | |  |             | |  |          |
  +------+      |  +----------+ |  |     | |  +-------------+ |  +----------+
                |               |  | upt | |                  |
  +------+      |  +----------+ |  |     | |  +-------------+ |  +----------+
  |      |      |  |          | |  |     | |  |             | |  |          |
  | CPAN |------|->| upt-cpan |-|->|     |-|->| upt-openbsd |-|->| Makefile |
  |      |      |  |          | |  |     | |  |             | |  |          |
  +------+      |  +----------+ |  +-----+ |  +-------------+ |  +----------+
                |               |          |                  |
```

Here we can see that using the [upt-pypi](https://pypi.python.org/pypi/upt-pypi)
and [upt-cpan](https://pypi.python.org/pypi/upt-cpan) frontends, upt is able to
retrieve information about packages hosted on [CPAN](https://www.cpan.org/) and
[PyPI](https://pypi.python.org/). Then, using the
[upt-guix](https://pypi.python.org/pypi/upt-guix) backend, it can generate a
package definition suitable for [GNU Guix](https://www.gnu.org/s/guix); using
the [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) backend, it can
generate a Makefile suitable for [OpenBSD](https://www.openbsd.org)'s ports.

Using this modular approach, we solved our two issues:
151

Cyril Roelandt's avatar
Cyril Roelandt committed
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
- We will only ever need *one* PyPI parser, *one* CPAN parser, etc.
- We will only have one backend per distribution. Backends will not be truly
  language-agnostic: the build systems will not be the same for Python packages
  and Ruby packages, for instance. Still, a lot of code may be factorized: the
  creation of the required directories, the writing of some common metadata...

Last but not least, frontends and backends are not part of the upt core project.
Every backend, every frontend is a separate project, living in its own
repository, written by its own team. This way, nobody has control over the full
project, and different communities may make different technical choices (testing
tools, coding standards, version control systems...) without having to impose
them upon everyone else.


## Contributing to upt
You may contribute by sending patches to tipecaml@gmail.com or by opening a pull
request.

Your patches should not break anything, so, prior to sending them, be sure to
run:

    $ tox -eflake8
    $ tox -epy36

You may also want to contribute to one of the frontends/backends: they are
separate projects, so you should not send your patches to upt.


## Adding frontends or backends

182
183
184
185
186
187
188
189
190
191
192
193
### New frontend/backend using cookiecutter

You may create the whole directory structure for a new frontend/backend using
[cookiecutter](https://github.com/audreyr/cookiecutter). This will also write
some code and create a first git commit including everything you need to get
started. Simply type:

    $ cookiecutter https://framagit.org/upt/cookiecutter-upt

And answer the questions prompted on the screen. The next two sections describe
the structure of a frontend and a backend.

Cyril Roelandt's avatar
Cyril Roelandt committed
194
### Adding a frontend
195
196
197
198
199
200
201
202
203
204
Your setup.cfg should use an entry point, like this:

```
# Taken from the upt-rubygems project
[options.entry_points]
upt.frontends =
    rubygems = upt_rubygems.upt_rubygems:RubyGemsFrontend
```

Alternatively, if you want to use setup.py:
Cyril Roelandt's avatar
Cyril Roelandt committed
205
206
207
208
209
210

```
setup(
    ...
    entry_points = { 
        'upt.frontends': [
211
            'rubygems=upt_rubygems.upt_rubygems:RubyGemsFrontend',
Cyril Roelandt's avatar
Cyril Roelandt committed
212
213
214
215
216
217
        ]   
    },
    ...
)
```

218
Let us now look at the implementation of a frontend:
Cyril Roelandt's avatar
Cyril Roelandt committed
219
220
221
222
223
224
225
226

```
...
import upt
...
class MyFrontEndPackage(upt.Package):
    pass

227

Cyril Roelandt's avatar
Cyril Roelandt committed
228
class MyFrontendParser(upt.Parser):
229
230
        name = 'myfrontend'

Cyril Roelandt's avatar
Cyril Roelandt committed
231
232
233
        def parse(self, pkg_name):
            ...
            return MyFrontendPackage(pkg_name,
234
235
236
237
238
239
240
241
242
                version=...,  # A string, such as '1.2.3'
                homepage=...,  # A string
                summary=...,  # A string containing a short description
                description=...,  # A string, containing a longer description
                download_urls=[url1, url2, ...],  # A list of strings
                # upt.PackageRequirement takes two arguments:
                # - the name of the package
                # - a version specifier, as defined in
                # https://www.python.org/dev/peps/pep-0440/#version-specifiers
Cyril Roelandt's avatar
Cyril Roelandt committed
243
                requirements={
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
                    'build': [
                         upt.PackageRequirement('build-dep', '>1.2'),
                         ...
                     ],
                    'run': [
                         upt.PackageRequirement('runtime-dep', ''),
                         ...
                     ],
                    'test': [
                         upt.PackageRequirement('test-dep', '<=2.3'),
                         ...
                     ]
                },
                # The license(s) used by this package, as instances of
                # upt.License. The upt project provides a lot of licenses in
                # the upt.licenses module. Use help(upt.licenses) to see the
                # whole list.
                # If you may not determine the nature of a licenses, you should
                # return upt.UnknownLicense().
                # The list of licenses may be empty.
                licenses=[
                    upt.licenses.BSDThreeClauseLicense(),
                    upt.UnknownLicense(),
Cyril Roelandt's avatar
Cyril Roelandt committed
267
268
269
270
                ],
                archives=[
                    upt.Archive('http://example.com/foo.tar.gz', size=1234,
                                md5='md5', rmd160='rmd160', sha256='sha256')
271
272
                ]
         )
Cyril Roelandt's avatar
Cyril Roelandt committed
273
274
275

```

276
277
278
279
The most interesting method here is MyFrontendParser.parse. It must return a
upt.Package object (you may define a subclass such as MyFrontendPackage if you
wish). Only the package name and its version must be passed to the constructor,
all other fields are optional.
Cyril Roelandt's avatar
Cyril Roelandt committed
280
281

### Adding a backend
282
283
284
285
286
287
288
289
290
291
292
Your setup.cfg should use an entry point, like this:

```
[options.entry_points]
# Taken from upt-openbsd
upt.backends =
    openbsd = upt_openbsd.upt_openbsd:OpenBSD
```

Alternatively, if you want to use setup.py:

Cyril Roelandt's avatar
Cyril Roelandt committed
293
294
295
296
297
```
setup(
    ...
    entry_points = { 
        'upt.backends': [
298
            'openbsd=upt_openbsd.upt_openbsd:OpenBSD',
Cyril Roelandt's avatar
Cyril Roelandt committed
299
300
301
302
303
304
        ]   
    },
    ...
)
```

305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
Let us now look at the implementation of a backend:

```
...
import upt
...
class MyBackend(upt.Backend):
    # The name of the backend: this is what will be seen in the output of
    # "upt --list-backends".
    name = 'mybackend'

    def create_package(self, upt_pkg, output=None):
        # Parameters:
        # - upt_pkg: an instance of upt_pkg. See the "Adding a frontend"
        #            section for more info on its fields.
        # - output: currently unused, will always be None.

        # Do whatever you need to generate a valid "package definition" for
        # your package manager. Note that "upt_pkg.frontend" contains the name
        # of the frontend that was used: it should be helpful with the
        # language-specific parts of your backend.
        ...

        # Currently, all backends write the "package definition" to stdout.
        print(...)
```

Cyril Roelandt's avatar
Cyril Roelandt committed
332

Cyril Roelandt's avatar
Cyril Roelandt committed
333
334
335
336
337
338
339
### Logging
Whether you are writing a frontend or a backend, you can use a logger provided
by upt by running:

```
import logging
...
Cyril Roelandt's avatar
Cyril Roelandt committed
340
logger = logging.getLogger('upt')
Cyril Roelandt's avatar
Cyril Roelandt committed
341
342
343
344
345
346
347
```

See the [logging module
documentation](https://docs.python.org/3/library/logging.html) for more info on
how to use this logger.


Cyril Roelandt's avatar
Cyril Roelandt committed
348
## Known frontends and backends
349
### Supported frontends
Cyril Roelandt's avatar
Cyril Roelandt committed
350
351
352
353
354

- [upt-cpan](https://pypi.python.org/pypi/upt-cpan) a frontend for
  [CPAN](https://www.cpan.org/)
- [upt-pypi](https://pypi.python.org/pypi/upt-pypi) a frontend for
  [PyPI](https://pypi.python.org/)
355
356
- [upt-rubygems](https://pypi.python.org/pypi/upt-rubygems) a frontend for
  [RubyGems](https://rubygems.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
357

358
359
360
Please tell us about your frontends!

### Supported backends
Cyril Roelandt's avatar
Cyril Roelandt committed
361

362
363
364
365
- [upt-fedora](https://pypi.python.org/pypi/upt-fedora) a backend for
  [Fedora](https://getfedora.org/)
- [upt-freebsd](https://pypi.python.org/pypi/upt-freebsd) a backend for
  [FreeBSD](https://www.freebsd.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
366
367
- [upt-guix](https://pypi.python.org/pypi/upt-guix) a backend for [GNU
  Guix](https://www.gnu.org/s/guix)
Cyril Roelandt's avatar
Cyril Roelandt committed
368
369
- [upt-nix](https://pypi.python.org/pypi/upt-nix) a backend for
  [Nix](https://nixos.org/nix/)
Cyril Roelandt's avatar
Cyril Roelandt committed
370
371
372
- [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) a backend for
  [OpenBSD](https://www.openbsd.org)

373
374
375
376
377
378
379
380
381
382
383
384
Please tell us about your backends!

### Compatibility

|             | upt-cpan | upt-pypi | upt-rubygems |
|-------------|----------|----------|--------------|
| upt-fedora  |    OK    |    OK    |      OK      |
| upt-freebsd |    OK    |    OK    |      KO      |
| upt-guix    |    OK    |    OK    |      OK      |
| upt-nix     |    OK    |    OK    |      KO      |
| upt-openbsd |    OK    |    OK    |      OK      |

Cyril Roelandt's avatar
Cyril Roelandt committed
385

386
387
## Similar projects

Cyril Roelandt's avatar
Cyril Roelandt committed
388
389
390
391
392
393
394
395
396
397
398
399
400
401
### Generic tools
[fpm](https://github.com/jordansissel/fpm) (Effing package management) is a tool
quite similar to upt. It can turn packages from PyPI/RubyGems/etc. into packages
for Debian/Fedora/etc. It features a unified command-line interface, a common
behaviour for all target distributions, and a modular design.

The main difference with upt is that it does not generate the "source" of a
package, but rather an installable package: it will generate a deb file rather
than a debian/ directory, an rpm file rather than a spec file, etc. See [this
comment on the bug
tracker](https://github.com/jordansissel/fpm/issues/1507#issuecomment-411390171)
for more information.

### Per-distribution tools
402
403
404
405
406
407
408
409
410
411
412
413
414
Here are some tools used to turn a package from a language-specific archive to
a package suitable for a general purpose package manager. We can see that [GNU
Guix](https://www.gnu.org/s/guix), [NetBSD](https://www.netbsd.org/) and
[OpenBSD](https://www.openbsd.org) decided to write a single tool, thus having
a unified CLI to access all upstream packages.

|      | Debian       | Fedora   | Guix        | FreeBSD  | NetBSD  | OpenBSD |
|------|--------------| ---------|-------------|----------|---------|---------|
| CPAN | dh-make-perl | cpan2rpm | guix import | ?        | url2pkg | PortGen |
| NPM  | npm2deb      | npm2rpm  | N/A         | ?        | url2pkg | ?       |
| PyPI | pypi2deb     | pyp2rpm  | guix import | pytoport | url2pkg | PortGen | 
| Ruby | gem2deb      | gem2rpm  | guix import | ?        | url2pkg | PortGen |

Cyril Roelandt's avatar
Cyril Roelandt committed
415
416
417
418

## License
This project is distributed under the [3-Clause BSD
License](https://opensource.org/licenses/BSD-3-Clause). See the LICENSE file.