ObsLocTAP < IVOA

IVOA Web>IvoaDAL>ObsLocTAP (2020-06-09, JesusSalgado) (raw view)
<h2 id="ImplementationguideofanObservationLocatorTAPservice-ObsLocTAPcomponents"> *ObsLocTAP project page* </h2>

This page summarises how to implement an ObsLocTAP service. The full description of this protocol can be found under the IVOA (International Virtual Observatory Alliance) domain at:

http://www.ivoa.net/documents/ObsLocTAP/index.html

The Observation Locator Table Access Protocol (ObsLocTAP henceforth) specifies in a standard format the services to retrieve information about planned, scheduled and performed observations. By sharing this information, the scientific community can monitor the planning activities of the observatories, prepare coordinated observations or to allow the preparation of better proposals from the scientific point of view.

Following the approach of the newer IVOA protocols, ObsLocTAP is a Tabular Access Protocol (TAP) service. This service exposes a database table using a query language close to SQL but with astronomical operators that allow better exploration of astronomical resources. This language called Astronomical Data Query Language (ADQL)

http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf

is a standard for the virtual observatory.

The implementation of a full compatible TAP service from scratch is cumbersome. However, there are a set of toolkits that allow the creation of this services without major knowledge of the underlying standards and very fast. We will describe how to implement a TAP service using a very well toolkit called taptuto but there are others. Implementors could take a look into the different alternatives and give them a try.

Other very well known toolkits are:
   * DACHS from GAVO: http://soft.g-vo.org/dachs https://dachs-doc.readthedocs.io/tutorial.html
   * SAADA: http://saada.unistra.fr/saada/
   * TAPTuto: http://cdsportal.u-strasbg.fr/taptuto/
Points 3 and 5 are applicable for all the toolkits to create a table ObsLocTAP compliant.
<h2 id="ImplementationguideofanObservationLocatorTAPservice-ObsLocTAPcomponents"> *ObsLocTAP COMPONENTS* </h2>

Three basic components are needed to implement an ObsLocTAP service
   * *Database installation*: In order to facilitate the geometrical queries needed for a TAP service, we recommend the use of a PostgreSQL DB instance
   * *Creation of ObsLocTAP complaint table (database)*
   * *TAP service instance*: Several frameworks can be used to create a TAP service. We will use, as example, the one created by Gregory Mantelet, currently working at CDS, due to its level of documentation. This package is open source
   * *Publication of tables through TAP server (TAP_SCHEMA)*
<h2 id="ImplementationguideofanObservationLocatorTAPservice-Databaseinstallation"> *DATABASE INSTALLATION* </h2>

There are several pages where we can find how to install a PosgreSQL server. In most of the cases, just following the instructions of https://www.postgresql.org/download/ should be enough to have a PosgreSQL server up and running. In fact, many of the different server images have already a PosgreSQL instance that can be used. Although PosgreSQL v10 contains some interesting features, v9.6 can guarantee at this stage operational performance and compatibility with extensions. There are multiple tutorials to install PosgreSQL depending on the operating system on the web.

It is recommendable to implement periodic dumps and backups in order to restore an operational database in case of failure (see https://www.postgresql.org/docs/9.6/backup.html). Also, it is recommendable to use a special account (e.g. postgres) for the database instance.

Once you have your PosgreSQL instance up and running, there is one spatial extension that would be needed for the TAP service instance. There are other spatial indexing extensions but the TAP service instance used in point 2 is making use of pgsphere.

Installation of the extension following the instructions at http://pgsphere.github.io/download.html is more or less transparent.
<h2 id="ImplementationguideofanObservationLocatorTAPservice-CreationofObsLocTAPcomplainttable(database)"> *CREATION OF OBSLOCTAP COMPLAINT TABLE (DATABASE)* </h2>

Create a database and a table with the following structure:
<pre>CREATE SCHEMA ivoa;
CREATE TABLE ivoa.obsplan (
  t_planning    double precision NOT NULL,
  target_name   character varying,
  obs_id      character varying NOT NULL,
  obs_collection  character varying NOT NULL,
  s_ra       double precision,
  s_dec      double precision,
  s_fov      double precision,
    s_region          scircle,
  s_resolution   double precision,
  t_min      double precision NOT NULL,
  t_max      double precision NOT NULL,
  t_exptime    double precision,
  t_resolution   double precision,
  em_min      double precision,
  em_max      double precision,
  em_res_power   double precision,
  o_ucd      character varying NOT NULL,
  pol_states    character varying,
  pol_xel     integer,
  facility_name  character varying NOT NULL,
  instrument_name character varying NOT NULL,
  obs_release_date timestamp,
  t_plan_exptime  double precision NOT NULL,
  category     character varying,
  priority     integer,
  execution_status character varying NOT NULL
);
CREATE INDEX i_obsplan_fov
 ON ivoa.obsplan
 USING gist
 (s_region);
</pre>

Please notice that the fov is defined as a pgsphere object. Other types different than polygon could be used (see https://pgsphere.github.io/doc/types.html) (a typical type used for footprints is circle) but two different types of pgsphere objects cannot be combined in the same database object. The last create index sentence defines how to create a gist index on the pgsphere object to increase the performance.

In this case, we have defined s_region as a scircle pgsphere object. Typical value would be a spoly. However, there is not an easy way to mix pgsphere circles and polygons in the same column as they do not have a generic data type. One possible solution is to store everything as spoly and convert the circles in approximated polygons.

The output of a select on s_region is typically a string representation of the Field of View. The conversion from a pgsphere output to a STC-S output when this column is used on a SELECT condition is typically done by the TAP server.

For example, polygon is defined as:

Polygon J2000 158.408362 57.781849 158.501991 57.812066 158.558573 57.762136 158.465026 57.73196

where the points of the polygon is defined after J2000 in pairs of ra dec values.

Whenever there is some uncertainty on the final pointing, a good option is to define the stc_s representation just as a circle, with the central position and the radius defined as:

Circle J200 &lt;s_ra&gt; &lt;s_dec&gt; &lt;s_fov&gt;

where the values of s_ra, s_dec and s_fov are the other columns in the obsplan table. Whenever the Field of View is better defined, an upgrade from the circle to something more detailed could be done.

See more info at http://ivoa.net/documents/STC-S/

It would be a good idea to populate this table with some dummy rows for testing during next steps.

If you are familiar with docker, we have created a docker instance with posgreSQL, pgsphere and the ivoa.obsplan table with some dummy fields for testing.

https://hub.docker.com/r/jsalgadodocker/postgres9.5-pgsphere-obsplan
<h2 id="ImplementationguideofanObservationLocatorTAPservice-TAPServiceinstance"> *TAP SERVICE INSTANCE* </h2>

There are some frameworks to be used to create a TAP instance. We recommend to use the one created by Gregory Mantelet at CDS as it is quite well documented and it has been updated to the latests ADQL (Astronomical Data Query Language) changes.

Installation instructions can be found here: http://cdsportal.u-strasbg.fr/taptuto/

Some notes:
   * ObsLocTAP database could/should be named *ivoa* instead of the one mentioned into the example *MyStarCatalogues* to clarify the schema
   * Table to be published will be called *obsplan*
The instructions provided by these pages are long but very descriptive so, in principle, you should be able to have a service running at the end of this process.
<h2 id="ImplementationguideofanObservationLocatorTAPservice-PublicationoftablesthroughTAPserver"> *PUBLICATION OF TABLES THROUGH TAP SERVER* </h2>

TAP_SCHEMA should contain the description of the published table(s) through the TAP service. Using the SQL script provided in the previous framework (or by other method), the characterization of the table in the TAP_SCHEMA should contain the following information:

| ivoa.obsplan | t_planning | adql:DOUBLE | d |
| ivoa.obsplan | target_name | adql:VARCHAR | |
| ivoa.obsplan | obs_id | adql:VARCHAR | |
| ivoa.obsplan | obs_collection | adql:VARCHAR | |
| ivoa.obsplan | s_ra | adql:DOUBLE | deg |
| ivoa.obsplan | s_dec | adql:DOUBLE | deg |
| ivoa.obsplan | s_fov | adql:DOUBLE | deg |
| ivoa.obsplan | s_resolution | adql:DOUBLE | arcsec |
| ivoa.obsplan | t_min | adql:DOUBLE | d |
| ivoa.obsplan | t_max | adql:DOUBLE | d |
| ivoa.obsplan | t_exptime | adql:DOUBLE | s |
| ivoa.obsplan | t_resolution | adql:DOUBLE | s |
| ivoa.obsplan | em_min | adql:DOUBLE | m |
| ivoa.obsplan | em_max | adql:DOUBLE | m |
| ivoa.obsplan | em_res_power | adql:DOUBLE | |
| ivoa.obsplan | o_ucd | adql:VARCHAR | |
| ivoa.obsplan | pol_states | adql:VARCHAR | |
| ivoa.obsplan | pol_xel | adql:BIGINT | |
| ivoa.obsplan | facility_name | adql:VARCHAR | |
| ivoa.obsplan | instrument_name | adql:VARCHAR | |
| ivoa.obsplan | obs_release_date | adql:TIMESTAMP | date |
| ivoa.obsplan | t_plan_exptime | adql:DOUBLE | s |
| ivoa.obsplan | category | adql:VARCHAR | |
| ivoa.obsplan | priority | adql:INTEGER | |
| ivoa.obsplan | fov | adql:VARCHAR | |
| ivoa.obsplan | stc_s | adql:VARCHAR | |

<h2 id="ImplementationguideofanObservationLocatorTAPservice-Testingthesystem"> *TESTING THE SYSTEM* </h2>

<span style="background-color: transparent;">Once the service is up and running, some queries can be used to test the system. Some examples are included in the specification but, as an example, we present some queries used for the </span><span style="background-color: transparent;">ObsLocTAP</span><span style="background-color: transparent;"> Integral Observatory service. The service URL for this service is:</span>

[[https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=select+o.s_ra,+o.s_dec,+o.target_name,++o.t_min,+o.t_max+from+ivoa.obsplan+as+o+where+o.t_max&gt;56738+order+by+o.t_min][https://iladev.esac.esa.int/tap-dev/tap/sync?]]

For other services, this service URL will be different although the rest of parameters (REQUEST, PHASE, LANG, QUERY) are the typical TAP service one so these will be common for the rest of services.
<h3 id="ImplementationguideofanObservationLocatorTAPservice-Searchofobservationsonacertaintimerange">6.1. SEARCH OF OBSERVATIONS ON A CERTAIN TIME RANGE</h3>

_ADQL Query:_
<pre>SELECT o.s_ra, o.s_dec, o.target_name, o.t_min, o.t_max
FROM ivoa.obsplan AS o
WHERE o.t_max &gt; 56783 ORDER BY o.t_min</pre>

Service invocation:

[[https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=select+o.s_ra,+o.s_dec,+o.target_name,++o.t_min,+o.t_max+from+ivoa.obsplan+as+o+where+o.t_max&gt;56738+order+by+o.t_min][https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=select+o.s_ra%2C+o.s_dec%2C+o.target_name%2C++o.t_min%2C+o.t_max+from+ivoa.obsplan+as+o+where+o.t_max%3E56738+order+by+o.t_min]]
<h3 id="ImplementationguideofanObservationLocatorTAPservice-Searchofobservationsonacertainastronomicalarea">6SEARCH OF OBSERVATIONS ON A CERTAIN ASTRONOMICAL AREA</h3>

<span style="background-color: transparent;">ADQL Query:</span>
<pre>SELECT * FROM ivoa.obsplan WHERE
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))
</pre>

Service invocation:

<br /> [[http://service invocation: https//iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=SELECT * FROM ivoa.obsplan WHERE 1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))][https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=SELECT%20%2A%20FROM%20ivoa.obsplan%20WHERE%201%3DINTERSECTS%28s_region%2C%20CIRCLE%28%27ICRS%27%2C%20114.8251%2C%201.6179%2C%200.016666%29%29]]
<h3 id="ImplementationguideofanObservationLocatorTAPservice-Combinationoftimeandgeometricalconditions">COMBINATION OF TIME AND GEOMETRICAL CONDITIONS</h3>

ADQL Query:
<pre>SELECT * FROM ivoa.obsplan WHERE
t_max &lt; 58502 AND
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))
</pre>

Service invocation:

[[https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=][https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=SELECT%20%2A%20FROM%20ivoa.obsplan%20WHERE%20t_max%20%3C%2058502%20AND%201%3DINTERSECTS%28s_region%2C%20CIRCLE%28%27ICRS%27%2C%20114.8251%2C%201.6179%2C%200.016666%29%29]]
<h2 id="Playingwithdockers">7. *PLAYING WITH DOCKERS* </h2>

Although it is not recommended to use docker containers for an operational archive, we have defined two dockers containers for the DB and the TAP server that you can use to have a ObsLocTAP server up and running in minutes in you have the docker command line client installed.

You just need to:
<pre>docker pull jsalgadodocker/pgsphere-obsplan
docker pull jsalgadodocker/tapserver
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap &lt;tapserver_image_id&gt;
docker run -p 5432:5432 --net=db-network --name db  &lt;pgsphere-obsplan_image_id&gt;</pre>

You will have a TAP server with an obsplan table with some Integral records as test data running on your localhost. It could be tested by

http://localhost:8080/tap/tap/

Also, you can connect to port 5432 of your localhost with a dabase client (e.g. pgadmin3) to see the records or to ingest your own ones.

In order to make public your service, you can e.g. create a proxy pass redirect from an apache web server, use port 80 and open it to internet, etc.
Topic revision: r2 - 2020-06-09 - JesusSalgado
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema