README.md 16.6 KB
Newer Older
Cyril Roelandt's avatar
Cyril Roelandt committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Universal Packaging Tool (upt)

A unified CLI tool that converts a package from a language-specific package
manager (such as PyPI or NPM) to an almost ready-to-use package for Free
Unix-based operating systems (such as a GNU/Linux distribution or \*BSD).


## Installing
Please note that Python 3.6 or newer is required to run upt.

You may install upt from PyPI:

    $ pip install upt

You may also install it from the Git repository or a tarball:

17
    $ python setup.py install
Cyril Roelandt's avatar
Cyril Roelandt committed
18
19
20
21
22
23
24
25
26

By itself, upt is useless. It requires frontends and or backends to provide
useful features. See the list of available frontends/backends at the end of this
document. You may install them by running:

    $ pip install upt-pypi
    $ pip install upt-openbsd
    $ pip install ...

27
28
29
30
31
32
33
34
35
36
37
38
You may also install upt and all its frontends by running:

    $ pip install upt[frontends]

You may also install upt and all its backends by running:

    $ pip install upt[backends]

You may also install upt and all of its frontends and backends by running:

    $ pip install upt[frontends,backends]

Cyril Roelandt's avatar
Cyril Roelandt committed
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

## Usage
You may get help by running:

    $ upt --help

Upt gathers data about a package using a **frontend** and writes a package
definition using a **backend**. You may list all installed frontends and
backends by running:

```
$ upt list-frontends
cpan
pypi
$ upt list-backends
guix
openbsd
```

To package [the requests library from
PyPI](https://pypi.python.org/pypi/requests) for [GNU
Guix](https://www.gnu.org/s/guix), just write:

    $ upt package --frontend pypi --backend guix requests

Or, using shorter options:

    $ upt package -f pypi -b guix requests

68
69
70
71
You may also want to recursively package all of its requirements:

    $ upt package -f pypi -b guix -r requests

72
73
74
75
You may also want to package a version other than the latest one:

    $ upt package -f pypi -b guix requests@2.16.0

Cyril Roelandt's avatar
Cyril Roelandt committed
76
77
The package definition will be written to the standard output.

78
79
80
81
82
83
84
85
86
You may update an existing package to the latest version:

    $ upt package -f pypi -b guix -u requests

You may update an existing package to a specific version:

    $ upt package -f pypi -b guix -u requests@2.22.0


Cyril Roelandt's avatar
Cyril Roelandt committed
87
88
89
90
91

## Under the hood
There are two kinds of places from where packages may be installed:

- language specific package repositories (CPAN, CRAN, CTAN, Hackage, NPM, PyPI,
92
  RubyGems, etc.)
Cyril Roelandt's avatar
Cyril Roelandt committed
93
- traditional, language agnostic package managers (apt for Debian, dnf for
94
  Fedora, Nix, Guix, pkgsrc, the OpenBSD ports, etc.)
Cyril Roelandt's avatar
Cyril Roelandt committed
95
96
97
98
99
100

Packages available on the former are usually also packaged in the latter ones.
Since packaging is a tedious and repetitive task, packagers have written tools
to assist them in this endeavor: Guix has "guix import", a collection of scripts
that parse CPAN, PyPI and other language specific package repositories and
generate a Guix package definition; Debian has tools such as pypi2deb,
101
102
npm2deb... The same goes for RPM-based distributions, and other operating
systems.
Cyril Roelandt's avatar
Cyril Roelandt committed
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167

Let's talk about PyPI. It seems that (almost) every single distribution has (or
will have) a tool that parses PyPI. Whatever the distribution may be (Debian,
Fedora, Guix...), parsing PyPI is basically always the same thing: it is just
reading a JSON document. This code is not shared among distributions though,
which means that they all end up writing the same kind of code. The same is true
for other language specific package managers.

The distribution specific code is usually not even shared among the various
tools provided by said distribution. Writing an OpenBSD port always involve
creating a directory, writing the maintainer's name to a Makefile, etc. These
operations are common to all packages: they should not be rewritten over and
over by every single tool OpenBSD developers may end up using. The same is true
for Debian: creating a Debian package involves a lot of operations that are not
truly specific to the piece of software being packaged.

We may also note that all these tools come with their own interface: packagers
working with more than one package repository and/or more than one distribution
therefore have to use different interfaces to perform similar tasks.

The Universal Packaging Tool (upt) aims at providing package maintainers with a
unified CLI that harnesses the power of a modular infrastructure, meant to
factorize as much code as possible. Upt is the "core" of the project. It exposes
a CLI and a small Python library that defines an internal representation for
packages. Two kind of objects have to talk to upt:

- frontends, which parse a package repository and return information about a
  given package using upt's internal representation;
- backends, which turn the internal representation of a package into a valid
  package definition for a language agnostic package manager.

This is basically what other common tools do: the [GCC
compiler](https://gcc.gnu.org/) has frontends for languages and backends for CPU
architectures. Similarly, [pandoc](https://pandoc.org/) can convert files from
one markup language to another using various frontends and backends.

```
                |               |          |                  |
   Package      |      upt      |    upt   |       upt        |    Package
 Repositories   |   Frontends   |          |     Backends     |  Definitions
                |               |          |                  |
  +------+      |  +----------+ |  +-----+ |  +-------------+ |  +----------+
  |      |      |  |          | |  |     | |  |             | |  |          |
  | PyPI |------|->| upt-pypi |-|->|     |-|->|  upt-guix   |-|->| Guix pkg |
  |      |      |  |          | |  |     | |  |             | |  |          |
  +------+      |  +----------+ |  |     | |  +-------------+ |  +----------+
                |               |  | upt | |                  |
  +------+      |  +----------+ |  |     | |  +-------------+ |  +----------+
  |      |      |  |          | |  |     | |  |             | |  |          |
  | CPAN |------|->| upt-cpan |-|->|     |-|->| upt-openbsd |-|->| Makefile |
  |      |      |  |          | |  |     | |  |             | |  |          |
  +------+      |  +----------+ |  +-----+ |  +-------------+ |  +----------+
                |               |          |                  |
```

Here we can see that using the [upt-pypi](https://pypi.python.org/pypi/upt-pypi)
and [upt-cpan](https://pypi.python.org/pypi/upt-cpan) frontends, upt is able to
retrieve information about packages hosted on [CPAN](https://www.cpan.org/) and
[PyPI](https://pypi.python.org/). Then, using the
[upt-guix](https://pypi.python.org/pypi/upt-guix) backend, it can generate a
package definition suitable for [GNU Guix](https://www.gnu.org/s/guix); using
the [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) backend, it can
generate a Makefile suitable for [OpenBSD](https://www.openbsd.org)'s ports.

Using this modular approach, we solved our two issues:
168

Cyril Roelandt's avatar
Cyril Roelandt committed
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
- We will only ever need *one* PyPI parser, *one* CPAN parser, etc.
- We will only have one backend per distribution. Backends will not be truly
  language-agnostic: the build systems will not be the same for Python packages
  and Ruby packages, for instance. Still, a lot of code may be factorized: the
  creation of the required directories, the writing of some common metadata...

Last but not least, frontends and backends are not part of the upt core project.
Every backend, every frontend is a separate project, living in its own
repository, written by its own team. This way, nobody has control over the full
project, and different communities may make different technical choices (testing
tools, coding standards, version control systems...) without having to impose
them upon everyone else.


## Contributing to upt
You may contribute by sending patches to tipecaml@gmail.com or by opening a pull
request.

Your patches should not break anything, so, prior to sending them, be sure to
run:

    $ tox -eflake8
    $ tox -epy36

You may also want to contribute to one of the frontends/backends: they are
separate projects, so you should not send your patches to upt.


## Adding frontends or backends

199
200
201
202
203
204
205
206
207
208
209
210
### New frontend/backend using cookiecutter

You may create the whole directory structure for a new frontend/backend using
[cookiecutter](https://github.com/audreyr/cookiecutter). This will also write
some code and create a first git commit including everything you need to get
started. Simply type:

    $ cookiecutter https://framagit.org/upt/cookiecutter-upt

And answer the questions prompted on the screen. The next two sections describe
the structure of a frontend and a backend.

Cyril Roelandt's avatar
Cyril Roelandt committed
211
### Adding a frontend
212
213
214
215
216
217
218
219
220
221
Your setup.cfg should use an entry point, like this:

```
# Taken from the upt-rubygems project
[options.entry_points]
upt.frontends =
    rubygems = upt_rubygems.upt_rubygems:RubyGemsFrontend
```

Alternatively, if you want to use setup.py:
Cyril Roelandt's avatar
Cyril Roelandt committed
222
223
224
225
226
227

```
setup(
    ...
    entry_points = { 
        'upt.frontends': [
228
            'rubygems=upt_rubygems.upt_rubygems:RubyGemsFrontend',
Cyril Roelandt's avatar
Cyril Roelandt committed
229
230
231
232
233
234
        ]   
    },
    ...
)
```

235
Let us now look at the implementation of a frontend:
Cyril Roelandt's avatar
Cyril Roelandt committed
236
237
238
239
240
241
242
243

```
...
import upt
...
class MyFrontEndPackage(upt.Package):
    pass

244

Cyril Roelandt's avatar
Cyril Roelandt committed
245
class MyFrontendParser(upt.Parser):
246
247
        name = 'myfrontend'

Cyril Roelandt's avatar
Cyril Roelandt committed
248
249
250
        def parse(self, pkg_name):
            ...
            return MyFrontendPackage(pkg_name,
251
252
253
254
255
256
257
258
                version=...,  # A string, such as '1.2.3'
                homepage=...,  # A string
                summary=...,  # A string containing a short description
                description=...,  # A string, containing a longer description
                # upt.PackageRequirement takes two arguments:
                # - the name of the package
                # - a version specifier, as defined in
                # https://www.python.org/dev/peps/pep-0440/#version-specifiers
Cyril Roelandt's avatar
Cyril Roelandt committed
259
                requirements={
260
261
262
263
                    'config': [
                         upt.PackageRequirement('config-dep', '==1.4'),
                         ...
                     ],
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
                    'build': [
                         upt.PackageRequirement('build-dep', '>1.2'),
                         ...
                     ],
                    'run': [
                         upt.PackageRequirement('runtime-dep', ''),
                         ...
                     ],
                    'test': [
                         upt.PackageRequirement('test-dep', '<=2.3'),
                         ...
                     ]
                },
                # The license(s) used by this package, as instances of
                # upt.License. The upt project provides a lot of licenses in
                # the upt.licenses module. Use help(upt.licenses) to see the
                # whole list.
                # If you may not determine the nature of a licenses, you should
                # return upt.UnknownLicense().
                # The list of licenses may be empty.
                licenses=[
                    upt.licenses.BSDThreeClauseLicense(),
                    upt.UnknownLicense(),
Cyril Roelandt's avatar
Cyril Roelandt committed
287
288
289
290
                ],
                archives=[
                    upt.Archive('http://example.com/foo.tar.gz', size=1234,
                                md5='md5', rmd160='rmd160', sha256='sha256')
291
292
                ]
         )
Cyril Roelandt's avatar
Cyril Roelandt committed
293
294
295

```

296
297
298
299
The most interesting method here is MyFrontendParser.parse. It must return a
upt.Package object (you may define a subclass such as MyFrontendPackage if you
wish). Only the package name and its version must be passed to the constructor,
all other fields are optional.
Cyril Roelandt's avatar
Cyril Roelandt committed
300
301

### Adding a backend
302
303
304
305
306
307
308
309
310
311
312
Your setup.cfg should use an entry point, like this:

```
[options.entry_points]
# Taken from upt-openbsd
upt.backends =
    openbsd = upt_openbsd.upt_openbsd:OpenBSD
```

Alternatively, if you want to use setup.py:

Cyril Roelandt's avatar
Cyril Roelandt committed
313
314
315
316
317
```
setup(
    ...
    entry_points = { 
        'upt.backends': [
318
            'openbsd=upt_openbsd.upt_openbsd:OpenBSD',
Cyril Roelandt's avatar
Cyril Roelandt committed
319
320
321
322
323
324
        ]   
    },
    ...
)
```

325
326
327
328
329
330
331
332
333
334
335
Let us now look at the implementation of a backend:

```
...
import upt
...
class MyBackend(upt.Backend):
    # The name of the backend: this is what will be seen in the output of
    # "upt --list-backends".
    name = 'mybackend'

336
337
338
339
340
341
342
343
344
345
346
347
348
    # Should you wish to use the "--recursive" flag, in order to recursively
    # package all dependencies of your package, you must define the following
    # method, which returns a list of all versions of a given package currently
    # packaged in your backend.
    def package_versions(self, package_name):
        """Return a list of available versions of PACKAGE_NAME"""
        raise NotImplementedError

    # Alternatively, you could redefine the following method to have complete
    # control over the decision to package (or not package) the given
    # requirement:
    # def needs_requirement(self, req, phase):

349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
    def create_package(self, upt_pkg, output=None):
        # Parameters:
        # - upt_pkg: an instance of upt_pkg. See the "Adding a frontend"
        #            section for more info on its fields.
        # - output: currently unused, will always be None.

        # Do whatever you need to generate a valid "package definition" for
        # your package manager. Note that "upt_pkg.frontend" contains the name
        # of the frontend that was used: it should be helpful with the
        # language-specific parts of your backend.
        ...

        # Currently, all backends write the "package definition" to stdout.
        print(...)
```

Cyril Roelandt's avatar
Cyril Roelandt committed
365

Cyril Roelandt's avatar
Cyril Roelandt committed
366
367
368
369
370
371
372
### Logging
Whether you are writing a frontend or a backend, you can use a logger provided
by upt by running:

```
import logging
...
Cyril Roelandt's avatar
Cyril Roelandt committed
373
logger = logging.getLogger('upt')
Cyril Roelandt's avatar
Cyril Roelandt committed
374
375
376
377
378
379
380
```

See the [logging module
documentation](https://docs.python.org/3/library/logging.html) for more info on
how to use this logger.


Cyril Roelandt's avatar
Cyril Roelandt committed
381
## Known frontends and backends
382
### Supported frontends
Cyril Roelandt's avatar
Cyril Roelandt committed
383
384
385

- [upt-cpan](https://pypi.python.org/pypi/upt-cpan) a frontend for
  [CPAN](https://www.cpan.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
386
387
- [upt-cran](https://pypi.python.org/pypi/upt-cran) a frontend for
  [CRAN](https://cran.r-project.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
388
389
- [upt-pypi](https://pypi.python.org/pypi/upt-pypi) a frontend for
  [PyPI](https://pypi.python.org/)
390
391
- [upt-rubygems](https://pypi.python.org/pypi/upt-rubygems) a frontend for
  [RubyGems](https://rubygems.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
392

393
394
395
Please tell us about your frontends!

### Supported backends
Cyril Roelandt's avatar
Cyril Roelandt committed
396

397
398
399
400
- [upt-fedora](https://pypi.python.org/pypi/upt-fedora) a backend for
  [Fedora](https://getfedora.org/)
- [upt-freebsd](https://pypi.python.org/pypi/upt-freebsd) a backend for
  [FreeBSD](https://www.freebsd.org/)
Cyril Roelandt's avatar
Cyril Roelandt committed
401
402
- [upt-guix](https://pypi.python.org/pypi/upt-guix) a backend for [GNU
  Guix](https://www.gnu.org/s/guix)
Cyril Roelandt's avatar
Cyril Roelandt committed
403
404
- [upt-nix](https://pypi.python.org/pypi/upt-nix) a backend for
  [Nix](https://nixos.org/nix/)
Cyril Roelandt's avatar
Cyril Roelandt committed
405
406
407
- [upt-openbsd](https://pypi.python.org/pypi/upt-openbsd) a backend for
  [OpenBSD](https://www.openbsd.org)

408
409
410
411
412
413
414
415
416
417
418
419
Please tell us about your backends!

### Compatibility

|             | upt-cpan | upt-pypi | upt-rubygems |
|-------------|----------|----------|--------------|
| upt-fedora  |    OK    |    OK    |      OK      |
| upt-freebsd |    OK    |    OK    |      KO      |
| upt-guix    |    OK    |    OK    |      OK      |
| upt-nix     |    OK    |    OK    |      KO      |
| upt-openbsd |    OK    |    OK    |      OK      |

Cyril Roelandt's avatar
Cyril Roelandt committed
420

421
422
## Similar projects

Cyril Roelandt's avatar
Cyril Roelandt committed
423
424
425
426
427
428
429
430
431
432
433
434
435
436
### Generic tools
[fpm](https://github.com/jordansissel/fpm) (Effing package management) is a tool
quite similar to upt. It can turn packages from PyPI/RubyGems/etc. into packages
for Debian/Fedora/etc. It features a unified command-line interface, a common
behaviour for all target distributions, and a modular design.

The main difference with upt is that it does not generate the "source" of a
package, but rather an installable package: it will generate a deb file rather
than a debian/ directory, an rpm file rather than a spec file, etc. See [this
comment on the bug
tracker](https://github.com/jordansissel/fpm/issues/1507#issuecomment-411390171)
for more information.

### Per-distribution tools
437
438
439
440
441
442
443
444
445
446
447
448
449
Here are some tools used to turn a package from a language-specific archive to
a package suitable for a general purpose package manager. We can see that [GNU
Guix](https://www.gnu.org/s/guix), [NetBSD](https://www.netbsd.org/) and
[OpenBSD](https://www.openbsd.org) decided to write a single tool, thus having
a unified CLI to access all upstream packages.

|      | Debian       | Fedora   | Guix        | FreeBSD  | NetBSD  | OpenBSD |
|------|--------------| ---------|-------------|----------|---------|---------|
| CPAN | dh-make-perl | cpan2rpm | guix import | ?        | url2pkg | PortGen |
| NPM  | npm2deb      | npm2rpm  | N/A         | ?        | url2pkg | ?       |
| PyPI | pypi2deb     | pyp2rpm  | guix import | pytoport | url2pkg | PortGen | 
| Ruby | gem2deb      | gem2rpm  | guix import | ?        | url2pkg | PortGen |

Cyril Roelandt's avatar
Cyril Roelandt committed
450
451
452
453

## License
This project is distributed under the [3-Clause BSD
License](https://opensource.org/licenses/BSD-3-Clause). See the LICENSE file.