*DAL 2 Date: Friday 6 June 2025 Time: 9:00-10:30 EDT Moderator: Joshua Fraustro Notetaker: Grégory Mantelet Participants: Joshua Fraustro (JF), Stephane Erard (SE), Jose Osinde Lopez (JOL), Pat Dowler (PD), Grégory Mantelet (GM), Markus Demleitner (MD), Mark Taylor (MT), Alberto Micol (AM), ... -------------------------------------------------------------------------------------------------------------------------------------- *Accessing Solar System Observations From ESO (Stephane Erard) * Provide Users with easy access to planetary data in ESO archives * Scope: planets, dwarf planets, satellites, interpla. medium * Science data only, possibly calibr. w/ automatic pipeline * EPNCore table at ESO * ESO TAP access: * Usual issue with supporting moving targets, * Targets are identified through SIMBAD only * raw data search interface handy, but limited in # of answers * 2 tables: * Obscore (moving targets) * dbo.raw: reduced and raw data * The only way to search by targets: main target name (base don OB parameter) * Test with known objecst ; no attempt to identify secondary targets * Identify target name / class during query * Mars in ObsCore provides > 1k results * Mars in dbo. raw > 2.7k * Some program > 2010 in dbo not in ObsCore at all? * Problems: * No indication of science field: SSO are not tagged (as in CAOM / HST) * Would make search by target name much easier * Care should be taken to retrieve target names * => need to filter fale alerts/misleading names * ... * Setup of EPN-TAP service very easy * Prototype conversion of ESO table via DaCHS * Url of product / datalink look OK * Updates: * Feasible via TAP query on crontab * Need to check/fix ingestion script for new data * Add functionalities: * ephemeris * s_region * Possible Extensions: * Write phase angle and Ls in EPNCore table to use as search params (phase from ephemeris table) * Add disk center coordinates / attitude * Compute geometry, including secondary targets (like ESASky) * Tagging proposals to reduce false alarms * possible solution: (CAOM style) include in the table: proposal keywords, maybe AI (trainer on existing CAOM archives) * alternative: analyze frame content with Skybot (coord + time) * An epn_code table dedicated to Solar System objects would be a great addition * Conversion to EPN-TAP is out of reach-- hard to identify Solar System targets * Significant help to the planetary science in order to use ESO archives * publishing results from ESASky to EPN core would be a nice Q-AM: ESO will try to do his best. : We are aligned with the CADC system. This system can be used whenever you want. *Evolution of Stateless TAP (Jose Osinde Lopez) * TAP is a fundamental component of the ESDC archives * Preparing TAP+ for higher loads * robustness / fault tolerance for infrastrutcture / scalability / flexibility. * Data combination / format conversion add significant load to the TAP server itself * (these are functionalities added by ESDC) * Challenge: archives growing in size (e.g. Gaia, Euclid, ....) * missions getting bigger and bigger * need to redefine the solution (like compressing data) and hardware (CPU, memory, storage, ....) * increasing demand of users like: * combining data from different missions which is important and becomes harder and more complex with the increasing volume * combine data from different TAP services/archives so that being used in the same UI/client (e.g. ESASky) * each mission has its own resources (databases, services (TAP, DataLink, EPN-TAP, ...), ...) (see diagram) * Advantages of Stateless service: * Transparent for users * does not store any client session information between requests * requests are independant and self-contained * easier maintenance/upgrade * scalable * resilient and fault tolerant * better load balancing * improved performance * TAP+ is currently not stateless * tightly coupled to servlet container * need to identify sessions/users and requests (even for anonymous requests) * architecture functional but prevents horizontal scaling and load balancing * => goal: evolve TAP into a well integrated service in the ESA's ecosystem * Session handling: * TAP Library is now based on the Spring Framework and so Spring Session * session data not persisted in a shared relational database * pros: session sharing, session survival when the web-app restarts/crashes, easily deployable in other architecture (cloud) * cons: session read/write involves DB query (so, overhead compared to in-memory), DB scheme setup, ... * The idea is to move whatever is inside the TAP (memory) in an outside system * DB access: * need to review all databases * ... * Information about UWS events (jobs) have to be shared accross different instances * A TAP service must be easily configurable in multiple instance * Architecture and tests now more complex in order to cover all use cases even though it is more robust *Management of Tap Uploads (Markus Demleitner) * Topic previously mentioned in Malta interop * Uploads are done using HTTP-PUT requests (/tap_user/) * or CREATE TABLE syntax * 4 open points available (see QRCode on slides) * deferring destruction (aka cleaning up tables after some defined time) * feature of UWS * problem: what if you need the table for longer? * solution: use UWS-like feature to post a new destruction date thanks to the DESTRUCTION parameter (simple curl command) * not really like this solution, but UWS proposes this solution anyway * rejected alternative: posting the new destruction date to the table URL directly instead of a child (i.e. /destruction) * see rejection reasons on slides * creating indexes * users sometimes want to create index to optimise the query execution (e.g. names, ids, positions) * any dbms like postgres provides a syntax to create index: CREATE INDEX ... * solution: `curl -F INDEX=Kmag http://.....user_tables/my_upload/index` * HTTP-GET on the /index endpoint gives debug information (supposed ot be human readable) * a problem arises: spatial indexes (useful for search on the sky with the distance(...) function for instance) * you need spatial indexes (general provided by extensions like q3c, pgsphere) * the server should automatically guess the type of index * best effort * open issues: * indexing a (user)table is an async task ; by default to QUEUED (PENDING if needed to be editable) * does this job to be destroyed automatically at its end? Or with a short-destruction time? * spatial or "normal" index when a user asks for (gal long, dec) * are column-references case-sensitive? * what about delimited identifiers? * Question: should MD open a Pull Request with this proposal? * See also a blog post (see QRCode on the slides) Q-JS: It is really close to what TAP+ is doing. Not yet used in-depth indexing. CADC has already persistent uploads MD: someone from esac has rasied a consensus in Malta interop. MT: How do you know when the job has been created if the job creating the index disappeared? MD: that's a problem. That's why immediate destruction is a bad idea. But even though it is not destroyed immediately don't expect it be here for long. GL: Really interesting job. Have you invetsigated an alternative to the API to access the indexes? Like an additional table than TAP_SCHEMA? So that you can have other information like stats MD: Not wild against it. THe content of this schema would be different for each user. Not impossible, but a lot of work. A VOSI table would be a good way to discover that kind of data. JS: In TAP+, the TAP_SHEMA is extended with the user data. This schema is regurlarly refreshed of course. *TAP-1.2 - OpenAPI and user-tables (Pat Dowler) * Update of work done of WD-TAP-1.2 and presented in the Malta interop * Mainly about user tables * Core changes for this are in VOSI ; while trying to use OpenAPI to describe it * Just created a minimal stub for some UWS components but it's incomplete * 4 yaml files for VOSI: * responses, - reusable components across a number of services * capabilities (+ HEAD and IAP responses), * tableset, * table (+ optional PUT, POST, DELETE ; to create, modify and drop tables) * Only caveat: some case sensitive parameters * Optional actions could use 405 (method not allowed) or 404 (not found) responses * 1st use case: As MD, async job to create indexes on persistent tables (issue #7 on VOSI repo) * Also using a UWS job to create index w/: table_name, column_name, optional UNIQUE flag (bool) * 2nd use case: creating permissions for user-managed tables (see issue #8 on VOSI repo) * permissions not needed for "private tables" * allow group permissions (see GMS) for project/collaboration * allow anonymous query (scenario: publish my result) * possible solutions: augment request/response in vosi-tables API, or another endpoint (implemented in Youcat) * UWS OpenAPI components in 3 yaml files: * responses * job * job-phase * more possible for child endpoints * TAP OpenAPI components in 6 yaml files: * responses * format * lang * query * async * the POST actions comes from TAP but the response is the normal UWS job listing * although UWS defines this part of the standard, the response codes special for TAP has to be defined here * sync * you have duplicated OpenAPI components * Since GET and POST are both allowed - same endpoint must be defined twice * ex: both async and sync reference the same OpenAPI doc for documents * The base endpoint is defined by the TAP document and the child paths are mostly described by the UWS one. * Dependence between UWS and TAP: * if the UWS OpenAPI component is augmented, the TAP one should be too * not ideal but no good alternative * Duplications are few time difficult to avoid * WD-TAP-1.2 is a work-in-progress * new features for user managed tables (WD-VOSI-1.2) * OpenAPI specification complete but not strict