Commit f5ba1d1b authored by Adrien Pavie's avatar Adrien Pavie
Browse files

Script for DB creation

parent e0dd0e86
......@@ -7,8 +7,24 @@ Osm4odf (OpenStreetMap pour OpenDataFrance) est un outil simple d'extraction th
* Et récupérer les données issues d'OpenStreetMap sur ce thème et sur ce territoire en CSV ou GeoJSON
## License
## Serveur
### Base de données
Vous devez disposer d'une base de données PostgreSQL dans une version récente (>= 9.6) avec les extensions suivantes installées :
* `hstore`
* `postgis`
### Configuration du script
Les configurations principales (accès à la base de données, dossier de travail, fichier PBF à récupérer...) sont à renseigner dans le fichier `db/run.sh`.
La configuration de la structure des données en bases (tags sous forme de colonnes, type d'objets sous forme d'emprise...) sont à détailler dans le fichier `db/import.style`. Un exemple complet de ce type de fichier [est disponible ici](https://github.com/openstreetmap/osm2pgsql/blob/master/default.style). Il est recommandé de ne faire apparaître sous forme de colonne dédiée uniquement les tags qui feront partie des filtres des requêtes formulées par l'API.
## Licence
Copyright (c) Adrien Pavie 2019
Released under AGPL license, see [LICENSE](LICENSE.txt) for complete license text.
Mis à disposition sous licence AGPL, voir [LICENSE](LICENSE.txt) pour le texte complet de la licence.
# This is the default osm2pgsql .style file that comes with osm2pgsql.
#
# A .style file has 4 columns that define how OSM objects end up in tables in
# the database and what columns are created. It interacts with the command-line
# hstore options.
#
# Columns
# =======
#
# OsmType: This is either "node", "way" or "node,way" and indicates if this tag
# applies to nodes, ways, or both.
#
# Tag: The tag
#
# DataType: The type of the column to be created. Normally "text"
#
# Flags: Flags that indicate what table the OSM object is moved into.
#
# There are 6 possible flags. These flags are used both to indicate if a column
# should be created, and if ways with the tag are assumed to be areas. The area
# assumptions can be overridden with an area=yes/no tag
#
# polygon - Create a column for this tag, and objects with the tag are areas
#
# linear - Create a column for this tag
#
# nocolumn - Override the above and don't create a column for the tag, but do
# include objects with this tag
#
# phstore - Same as polygon,nocolumn for backward compatibility
#
# delete - Drop this tag completely and don't create a column for it. This also
# prevents the tag from being added to hstore columns
#
# nocache - Deprecated and does nothing
#
# If an object has a tag that indicates it is an area or has area=yes/1,
# osm2pgsql will try to turn it into an area. If it succeeds, it places it in
# the polygon table. If it fails (e.g. not a closed way) it places it in the
# line table.
#
# Nodes are never placed into the polygon or line table and are always placed in
# the point table.
#
# Hstore
# ======
#
# The options --hstore, --hstore-match-only, and --hstore-all interact with
# the .style file.
#
# With --hstore any tags without a column will be added to the hstore column.
# This will also cause all objects to be kept.
#
# With --hstore-match-only the behavior for tags is the same, but objects are
# only kept if they have a non-NULL value in one of the columns.
#
# With --hstore-all all tags are added to the hstore column unless they appear
# in the style file with a delete flag, causing duplication between the normal
# columns and the hstore column.
#
# Special database columns
# ========================
#
# There are some special database columns that if present in the .style file
# will be populated by osm2pgsql.
#
# These are
#
# z_order - datatype int4
#
# way_area - datatype real. The area of the way, in the units of the projection
# (e.g. square mercator meters). Only applies to areas
#
# osm_user - datatype text
# osm_uid - datatype integer
# osm_version - datatype integer
# osm_changeset - datatype integer
# osm_timestamp - datatype timestamptz(0).
# Used with the --extra-attributes option to include metadata in the database.
# If importing with both --hstore and --extra-attributes the meta-data will
# end up in the tags hstore column regardless of the style file.
######################################################################
# Mandatory rules
# Do not change unless you know what you are doing
node,way boundary text polygon
node,way admin_level text polygon
node,way name text linear
node,way area text polygon
######################################################################
# Your own rules
# Only put tags that will be used in API SQL queries
# OsmType Tag DataType Flags
node,way leisure text polygon
node,way amenity text polygon
node,way building text polygon
node,way tourism text polygon
node,way landuse text polygon
node,way shop text polygon
node,way craft text polygon
node,way office text polygon
node,way emergency text polygon
node,way historic text polygon
node,way highway text linear
node,way access text linear
node,way wheelchair text linear
node,way cycleway text linear
node,way cycleway:both text linear
node,way cycleway:left text linear
node,way cycleway:right text linear
node,way addr:housenumber text polygon
#!/bin/bash
##############################################
# #
# Download and import raw OpenStreetMap data #
# Dependencies: #
# - Wget #
# - Psql #
# - Osm2PgSQL #
# #
##############################################
##########################################################################
#
# Configuration
# Please change values according to your server settings
#
# Work directory
# Defines where temporary files should be written on filesystem
WORK_DIR="/tmp/osm4odf"
# OpenStreetMap PBF extract
# You can choose the region you want on http://download.geofabrik.de/
PBF_URL="http://download.geofabrik.de/europe/france/bretagne-latest.osm.pbf" # "http://download.geofabrik.de/europe/france-latest.osm.pbf"
# PostgreSQL connection settings
# PG_DB_CLEAN is the name of final database (production)
# and PG_DB_TMP the name of next production database (to allow import in parallel)
PG_HOST="localhost"
PG_USER="postgres"
PG_PORT="5432"
PG_DB_CLEAN="osm4odf"
PG_DB_TMP="osm4odf_next"
PG_DB_OLD="osm4odf_prev"
# Parallel jobs
# You can set this value as 75% of your amount of CPU cores (and max 8)
PARALLEL_JOBS="6"
# Memory for import cache (in MB)
# You can set this as 75% of your server amount of RAM
CACHE_MEM="10000"
##########################################################################
#
# Script itself
# DO NOT CHANGE BELOW THIS (unless you know what you are doing) !
#
start=`date +%s`
# Create work directory if not exists
if ! mkdir -p "${WORK_DIR}"; then
echo "Can't create work directory"
exit 1
fi
# PBF extract download
last_ts_file="${WORK_DIR}/last_ts.txt"
yesterday_ts=`date +%s -d 'yesterday'`
pbf_file="${WORK_DIR}/extract.osm.pbf"
# We check how old is the latest extract to avoid over-downloading
if ! [ -f "${last_ts_file}" -a -r "${last_ts_file}" -a `cat "${last_ts_file}"` -gt "${yesterday_ts}" ]; then
# Start download
if wget "${PBF_URL}" -O "${pbf_file}"; then
# Success -> Write TS file
date +%s > "${last_ts_file}"
else
# Failure -> Stop script
echo "Can't download file"
rm "${pbf_file}" "${last_ts_file}"
exit 1
fi
else
echo "OSM extract recent enough, skip download"
fi
set -e
# Setup next database
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "DROP DATABASE IF EXISTS ${PG_DB_TMP}"
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "CREATE DATABASE ${PG_DB_TMP}"
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -d "${PG_DB_TMP}" -c "CREATE EXTENSION postgis; CREATE EXTENSION hstore"
# Start import
osm2pgsql --create \
-H "${PG_HOST}" -U "${PG_USER}" -P "${PG_PORT}" -d "${PG_DB_TMP}" \
--cache "${CACHE_MEM}" \
--number-processes "${PARALLEL_JOBS}" \
--style "import.style" \
--hstore --multi-geometry --keep-coastlines --extra-attributes \
"${pbf_file}"
# Add boundaries for API requests
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -d "${PG_DB_TMP}" -c "CREATE TABLE boundary AS SELECT osm_id, CASE WHEN boundary = 'administrative' THEN 'admin_'||admin_level ELSE boundary END AS type, name, way FROM planet_osm_polygon WHERE boundary IN ('administrative', 'local_authority', 'political') AND (admin_level IS NULL OR (admin_level::int >= 6 AND admin_level::int <= 8)); CREATE INDEX boundary_osm_id_idx ON boundary(osm_id); CREATE INDEX boundary_way_idx ON boundary USING GIST(way);"
# Alert API that database will be unavailable
echo "Stop API queries to update database"
lock_file="${WORK_DIR}/db.lock"
echo "lock" > "${lock_file}"
sleep 5
# Kill still running queries
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname='${PG_DB_CLEAN}'"
# Rename databases to make next goes live
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "DROP DATABASE IF EXISTS ${PG_DB_OLD}"
set +e
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "ALTER DATABASE ${PG_DB_CLEAN} RENAME TO ${PG_DB_OLD}"
set -e
psql -h "${PG_HOST}" -U "${PG_USER}" -p "${PG_PORT}" -c "ALTER DATABASE ${PG_DB_TMP} RENAME TO ${PG_DB_CLEAN}"
# Unlock API
rm "${lock_file}"
end=`date +%s`
echo "Done in $((end-start)) seconds"
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment