LargeSynopticSurveyTelescop
DataManagementSystem
K.-T.Lim,J.Bosch,G.Dubois-Felsmann,
W.O’Mullane,D.Petravick,andtheDMLeadership
LDM-148
LatestRevision:2017-07-04
ThisLSSTdocumenthasbeenapprovedasaContent-ControlledDocument
nicalControlTeam.Ifthisdocumentischangedorsuperseded,
theHandledesignationshownabove.Thecontrolisonthemost
thisHandleintheLSSTdigitalarchiveandnotprintedversions.
foundinthecorrespondingDMRFC.
LARGESYNOPTICSURVEYTELESCOPE
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
Abstract
TheLSSTDataManagementSystem(DMS)isasetofservices
ofsoftwarecomponentsrunningoncomputationalandnetworking
thatcombinetodeliversciencedataproductstotheobservatory’s
portobservatoryoperations.Thisdocumentdescribes
viceinstances,andtheirdeploymentenvironmentsas
them,therestoftheLSSTsystem,andtheoutsideworld.
Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
Team.
ii
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
ChangeRecord
VersionDateDescriptionOwnername
22011-08-09CopiedfromMREFCProposalintoLDM-148
handle,reformatted
RobertMcKercher
32011-08-15UpdatedforPreliminaryDesignReviewTimAxelrod,K-TLim,
MikeFreemon,Jeffrey
Kantor
42013-10-09UpdatedforFinalDesignReviewMarioJuric,K-TLim,Jef-
freyKantor
5.02017-07-04RewrittenforConstructionandOperations.
ApprovedinRFC-358.
K-TLim
Documentcurator:Kian-TatLim
https://github.com/lsst/LDM-148
The contents of this document are subject to configuration control by the
Team.
iii
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Contents
1 Introduction
1
2 Summary Concept of Operations
2
3 Sizing
4
4 Component Overview
8
5 Base Enclave
14
5.1 Service Descriptions
........................
16
5.1.1 Image and EFD
.....................
Archiving
16
5.1.2 Prompt Processing
.....................
Ingest
16
5.1.3 OCS Driven Batch
.....................
Control
17
5.1.4 Telemetry
.......................
Gateway
17
5.2 Interfaces
............................
17
6 NCSA Enclave Level 1 Domain
18
6.1 Service Descriptions
........................
19
6.1.1 Prompt Processing
.......................
19
6.1.2 OCS Driven Batch
....................
Processing
19
6.1.3 Offline Processing
.......................
20
6.1.4 Level 1 Quality
......................
Control
20
6.1.5 Alert Distribution
.......................
20
The contents of this document are subject to configuration control by the
Team.
iv
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
6.1.6 Alert........................
Filtering
20
6.2 Interfaces
............................
21
7 NCSA Enclave Level 2 Domain
21
7.1 Service Descriptions
........................
22
7.1.1 Template and Calibration Products
......
Production
22...
7.1.2 Data Release Production
..................
Execution 22
7.1.3 Level 2 Quality
......................
Control
23
7.2 Interfaces
............................
23
8 Satellite Computing Enclave
23
8.1 Service Description
.........................
23
8.1.1 Data Release Production
.........
Satellite Processing
23
.....
8.2 Interfaces
............................
23
9 Data Access Center Enclaves
24
9.1 Service Descriptions
........................
24
9.1.1 Bulk Data......................
Distribution
24
9.1.2 LSST Science Platform
.................
DAC instances24
9.2 Interfaces
............................
25
10 NCSA Enclave Analysis and Developer Support26
Domain
10.1 Service Descriptions
........................
26
10.1.1 LSST Science Platform Science
........
Validation
26
.....
instance
The contents of this document are subject to configuration control by the
Team.
v
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
10.1.2 Developer
.......................
Services
26
10.1.3 Integration
.....................
and Testing
26
10.2 Interfaces
............................
27
11 Commissioning Cluster
27
11.1 Service Description
.........................
27
11.1.1 LSST Science Platform.........
Commissioning
27
.....
instance
11.2 Interfaces
............................
28
12 Infrastructure Domain
28
12.1 Service Descriptions
........................
28
12.1.1 Data Backbone
........................
28
12.1.2 Management/Monitoring
.....................
30
12.1.3 Provisioning/Deployment
.....................
30
12.1.4 Workload/Workflow
......................
31
12.1.5 Batch Processing
.......................
31
12.1.6 Identity......................
Management
31
12.2 Interfaces
............................
31
13 Software Components
32
13.1 Science.........................
Payloads
32
13.1.1 Alert Production
.....................
Payload
32
13.1.2 MOPS.Payload
.......................
32
The contents of this document are subject to configuration control by the
Team.
vi
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
13.1.3 Raw Calibration..................
Validation Payload32
13.1.4 Daily Calibration Products
..........
Update Payload
33
......
13.1.5 Periodic Calibration Products
........33
Production
.....
Payload
13.1.6 Template Generation
...................
Payload
33
13.1.7 Annual Calibration Products
........33
Production
.....
Payload
13.1.8 Data Release Production
..................
Payload
33
13.2 SUIT
.............................
34
13.3 Middleware
...........................
34
13.3.1 Data Butler
.....................
Access Client
34
13.3.2 Parallel Distributed
.................
Database (Qserv)
34
13.3.3 Task Framework
.......................
34
14 Design and Implementation Standards
35
14.1 Python.Version
.........................
35
14.2 HTTPS Protocol
..........................
35
15 Appendix: Traceability
37
15.1 Requirement to Component
..................
Traceability 37
15.2 Component to Requirement
..................
Traceability 45
16 References
61
The contents of this document are subject to configuration control by the
Team.
vii
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Data Management System Design
1 Introduction
The purpose of the LSST Data Management System (DMS) is to
to the observatory’s users and to support observatory operations.
vices employing a variety of software components running
infrastructure. The DMS is constructed by the DM subsystem
Operations era, it is operated by a combination of the LSST
and Observatory Operations departments.
The data products to be delivered are defined and described
Document
LSE-163
(). These are divided into three major categories.
One category of data products is generated on a nightly or
processed/calibrated, and difference images as well as alerts
able objects detected from the images, published within
able catalogs. These data products can be considered “online”,
by the observing cadence of the observatory. This category
as “Level 1”. These products are intended to enable detection
time-domain events.
A second category of data products is generated on an ann
complete reprocessing of the set of images taken to date
containing measurements and characterization of tens of
high and uniform astrometric and photometric accuracy.
the first category of data products is regenerated, often
category also includes other data products such as calibration
are generated in an “offline” mode, not directly tied to the
has historically been referred to as “Level 2”, including
the first category.
The third category of data products is not generated by the
created, or imported by science users for their own science
value from their close association with or derivation from
The contents of this document are subject to configuration control by the
Team.
1
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
is responsible for providing facilities, services, and
This category has historically been referred to as “Level
Data products are delivered to science users through Data
streams of near-realtime alerts and telescope pointing
data product has associated metadata providing provenance
ing it to relevant calibration information in the archive.
but significant computational, storage, networking, and
a flexible, multi-tenant environment for professional astronomers
trieve, manipulate, and annotate LSST data products in order
and inquiry.
The first section of this document describes how the DMS components
erate and distribute the data products. The next section
computing environments was estimated. Subsequent sections
ponents of the DMS in more detail, including their interfaces
subsystems, and with the outside world.
2 Summary Concept of Operations
The principal functions of the DMS are to:
•Process the incoming stream of images generated by the
ing by archiving raw images, generating transient alerts,
and object catalogs.
•Periodically (at least annually) process the accumulated
photometric and astrometric calibration, measure the
characterize the time-dependent behavior of objects.
run form a data release (DR), which is a static, self-consistent
ing scientific analysis of LSST data and publication of
archived for the entire operational life of the LSST archive.
•Periodically create new calibration data products, such
be used by the other processing functions.
The contents of this document are subject to configuration control by the
Team.
2
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
•Make all LSST data available through an interface that
extent, community-based standards such as those being
servatory (VO) in collaboration with the International
Provide enough processing, storage, and network bandwidth
the data without petabyte-scale data transfers.
The latency requirements for alerts determine several aspects
cost. An alert is triggered by an unexpected excursion in
appearance of a previously undetected object such as a supernova
ical time scale of some of these events may warrant follow-up
time scales. These excursions in brightness must be recognized
sulting alert data product sent on its way, within 60 seconds.
the decision to acquire high-bandwidth/high-reliability
mit at Cerro Pachon to the Base in La Serena and from Chile
the significant computational resources necessary for promptly
to be located in cost-effective locations: the Base has far
and rack space capacity than the Summit, and placing the scientific
for far greater flexibility in the allocation of resources
forming cross-talk correction on the data in the data acquisition
alert processing at the amplifier and CCD levels, where possible,
latency to alert delivery.
The Data Release processing requires extensive computation,
images of an object in order to measure it as accurately as
and workflow management system and Task Framework are used
into manageable units of work that can be assigned to available
dedicated processing clusters at NCSA and CC-IN2P3.
Calibration data products must be created and updated at
Data Release periods. The stability of the system is expected
annual calibration productions. The daily production must
tory schedule, occurring after raw calibration frames have
observing is planned. This requirement necessitates the
Observatory Control System to trigger remote calibration
The DACs are a key component of the DMS, giving the community
The contents of this document are subject to configuration control by the
Team.
3
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
face to interact with and utilize the LSST data products
the LSST Science Platform (LSP) is deployed in each DAC to
tal, JupyterLab (notebook), and Web API aspects. Substantial
bandwidth is devoted to ensuring that the LSP is responsive
vast LSST data products.
Underlying all of the above is a Data Backbone that provides
for all LSST data products. The Data Backbone links all of
Data Access Centers, acting as the spine that supports them
3 Sizing
A fundamental question is how large the LSST Data Management
end, a complex analytical model has been developed driven
specifications. Specifications from the science requirements
and the observing strategy, translate directly into numbers
nomical objects, and ultimately into required network
systems. Specific science requirements of the survey determine
be maintained in the DMS products, which in turn determine
and the computer power necessary to execute them. The relationship
model and their flow-down from systems and DMS1.
requirements
De-
tailed sizing computations and associated explanations
the Figure.
Key input parameters include camera characteristics, the
the number of observed stars and galaxies expected per band,
data element, the data transfer rates between and within
query rates of input and output data, the alert generation
requirements for all data products.
Processing requirements were extrapolated from the functional
type pipelines and algorithms, and existing precursor pipelines
part of every Data Release, all data previously processed
gorithms, calibration products, and parameters. This causes
increase with time. Advances in hardware performance, however,
The contents of this document are subject to configuration control by the
Team.
4
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
DM Compute Sizing
LDM-138
Explanation:
LDM-140
DM Storage & I/O Sizing
LDM-141
Explanation:
LDM-139
Network Sizing
LDM-142
Explanation:
LSE-78
DM Site Specific Infrastructure
Architecture/Design/Costs
LDM-144
Explanation:
LDM-143
LSST Science Requirements
LPM-17
LSST System Requirements
LSE-29
Observatory System Specs
LSE-30
DM Subsystem Requirements
LSE-61
Data Products Definition Document
LSE-163
Community Access White Paper
Document-5373
LSST Science and
Project Sizing Inputs
LSE-81
Explanation:
LSE-82
DM System Design
LDM-148
DM Applications Design
LDM-151
DM Middleware Design
LDM-152
DM Database Design
LDM-135
DM Infrastructure Design
LDM-129
Figure1:DMSInfrastructureSizingandEstimation.
ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSSTDMTechnicalControl
Team.
5
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
Table1:DMSComputeInfrastructureSizing;growthfromSurvey
high-watermark
ArchiveSiteBaseSite
Compute
TeraFLOPS(sustained)197→97030→62
Nodes436→305(455hwm)56→17(59hwm)
Cores18K→62K3K→4K
Database
TeraFLOPS(sustained)40→31055→306
Nodes94→113(148hwm)108→98(133hwm)
Facilities
FloorSpace826→744ft
2(834hwm)278→195ft2(435hwm)
Power274→273kW(309hwm)158→248kW(248hwm)
Cooling0.9→0.9mmbtu(1.1hwm)0.5→0.8mmbtu(0.8hwm)
Table2:DMSStorageInfrastructureSizing;growthfromSurvey
ArchiveSiteBaseSite
FileStorage
Capacity24→81PB
Drives1602→862597→249
Bandwidth493→714GB/s(752hwm)223→231GB/s(236hwm)
Database
Capacity29→99PB16→72PB
Drives3921→22882190→1642
Bandwidth1484→2040GB/s(2163hwm)829→1169GB/s(1615hwm)
TapeStorage
Capacity31→242PB
Tapes2413→3691(4117hwm)
TapeBandwidth36→65GB/s
numberofnodesneededandthepowerandcoolingdevoted
theperformancefiguresinTable1toreachahigh-watermarkduringthesurvey.
Storageandinput/outputrequirementswereextrapolated
products,theDMSandprecursordatabaseschemas,andexisting
systemoverheadfactorsinprecursorsurveysandexperiments
summaryofkeynumbersisinTable2.
Communicationsrequirementsweredevelopedandmodeled
query/responseload,extrapolatedfromexistingsurveys
requirementsareillustratedinFigure2forper-visittransfers.Peakbandwidths
secondbudgetforSummittoBaseimagetransferanda5second
imagetransfer.
TheSummittoBaseandBasetoNCSAnetworklinkshavebeen
Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
Team.
6
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
forfourmainreasons:first,becausetheincrementalcosts
hasbeenprovisionedatallhavebeensmall;second,toallow
withtheCameraDataSystemorperformingimageanalysis
alerts,tobeperformedinappropriatelocations;third,
fourth,tosimplifycertaincomponentssuchastheForwarders
withordependonthenetworks.Ahigh-speedBasetoNCSA
tobetransferredsouthtotheChileanDACoverthenetwork
media,asoriginallyplanned,decouplingsciencefrommaintenance
agreaterextent.
Observatory
Prompt
Processing
Ingest
Image and
EFD
Archiving
Data
Backbone
Endpoint
Prompt
Processing
Alert
Distribution
Alert
Filtering
Camera Data
Raw
System (DAQ)
Images
24 GB
Crosstalk-
Corrected
Images
24 GB
Data
Backbone
Endpoint
Processed Visit Image
& Difference Image
36 GB each
Catalog Entries
72 MB
Alert Stream
1.3 GB
Full-Feed
Alert
Broker
Alert Stream
1.3 GB
Alert Client
Filtered
Alert Stream
1.3 MB
Image size estimates are per visit of two snap exposures,
uncompressed, either 18 bits per pixel transferred in 32 bits
or 3×32 bits per pixel (image, mask, variance). Alternate
Science Visits of one 30-second exposure would produce
one 12 GB raw image and one 12 GB crosstalk-corrected
image.
Alert stream and catalog estimates are for max
40K alerts per visit. Alerts include postage stamp images
and one year DIASource history, uncompressed.
Filtering ratio is for illustration only.
Base
NCSA
NCSA
External
Science
Users
LSST
Science
Platform
DAC
NCSA
Figure
Near
2:
Real-Time Data Flows
In all of the above, industry-provided
LDM-143
) were
technology
used to extrapolate
trends
to the LSST construction and operations phases in which the
figured, deployed, operated, and maintained. A just-in-time
to leverage favorable cost/performance trends.
The contents of this document are subject to configuration control by the
Team.
7
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
The resulting performance and sizing requirements show
class system with correspondingly large data input/output
spite this size, technology trends show this to be well within
commodity-based systems during the construction and operations
4 Component Overview
The services that make up the DMS are in turn made up of software
components, instantiated in a particular configuration in
to perform a particular function. Some software components
ers are general-purpose and reused across multiple services.
instance in the production system; others have several,
the development and integration environments for testing
The DMS services can be considered to consist of four tiers
tier is the LSST Science Platform, which is deployed in the
vironments to provide a user interface and analysis environment
staff. The detailed design of this tier
LDM-542
is given
. Thein
next
“LSST
tier
Science
is composed of science “applications” software that generates
is used to build “payloads”, sequences of pipelines, that
product generation tasks. It is also used by science users
ucts. The detailed design of the components in this tier is
PipelinesLDM-151
Design,
. A lower tier is “middleware” software components
that execute the science application payloads and isolate
cluding changes to underlying technologies. These components
science users and staff. The detailed design of the components
Management Middleware
LDM-152
.Design,
The bottom tier is “infrastructure”:
networking, and low-level software and services that provide
detailed design of components in this tier is given in Data
ture,
LDM-129
, and Network
LSE-78
Design,
.
The DMS computing environments reside in four main physical
including the main Observatory and Auxiliary Telescope
the Base Facility data center located at the Base Site in
data center at the National Center for Supercomputing Applications
The contents of this document are subject to configuration control by the
Team.
8
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
USA; and the Satellite Computing Facility at CC-IN2P3 in
high-speed networks to allow rapid data movement. The Base
production computational environments (the Base Enclave
and also the US and Chilean Data Access Centers. In addition,
putational environment also resides at the Base Facility.
The DMS service instances in the NCSA Enclave can be broken
domains: a near-realtime online domain (L1) closely linked
offline Level 2 domain (L2) organized primarily around the
and an analysis and developer support domain (ADS) encompassing
ations staff use for science validation, software development,
testing. In addition, an underlying infrastructure domain
of the other domains, including a common Data Backbone that
archiving and that is the primary connection between all
distinguished by having different users, operations timescales,
nents.
The service instances that make up the DMS include (with the
domains they are in noted):
•Image and EFD Archiving (Base)
•Prompt Processing Ingest (Base)
•Observatory Control System (OCS) Driven Batch Control
•Telemetry Gateway (Base)
•Prompt Processing (L1)
•OCS Driven Batch Processing (L1)
•Offline Processing (L1)
•Alert Distribution (L1)
•Alert Filtering (L1)
•Level 1 Quality Control (QC) (L1)
•Template and Calibration Products Production Execution
The contents of this document are subject to configuration control by the
Team.
9
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
•Data Release Production Execution (L2)
•Data Release Production Satellite Processing (Satellite
•Level 2 QC (L2)
•LSST Science Platform Commissioning Cluster instance
•LSST Science Platform Data Access Center instances (DACs)
•Bulk Data Distribution (DAC)
•LSST Science Platform Science Validation instance (ADS)
•Developer Services (ADS)
•Integration and Test (ADS)
•Data Backbone (Infra)
•Management/Monitoring(Infra)
•Provisioning/Deployment(Infra)
•Workload/Workflow(Infra)
•HTCondor Batch Processing (Infra)
•Identity Management (Infra)
The relationships between these services, their deployment
mains, and science application “payloads”
3.
can be visualized
The common infrastructure services
4.
are illustrated in Figure
The science application software for the Alert Production,
Production, and calibration processing is built out of a
In turn, those frameworks build on middleware that provides
relationships between the packages implementing these se
underlying middleware packages
5.
are shown in Figure
Key applications software components include:
The contents of this document are subject to configuration control by the
Team.
10
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Data Backbone
NCSA Enclave
Analysis and
Developer Support
Level 1
Level 2
Base Enclave
Prompt
Processing
Ingest
Offline
Processing
OCS
Driven
Batch Ctrl
Image and
EFD
Archiving
Level 1
Quality
Control
Alert
Distribution
Telemetry
Gateway
Alert
Filtering
Template &
Calib. Prod.
Production
Data
Release
Production
Level 2
Quality
Control
US Data Access Center
Bulk Data
Distribution
Science
Platform
(DAC)
Observatory
Developer
Services
Integration
& Test
Science
Platform
(Sci. Valid.)
Data
Backbone
Endpoint
Prompt
Processing
Data
Backbone
Endpoint
OCS Batch
Processing
Data
Backbone
Endpoint
Commissioning Cluster
Science
Platform
(Commiss.)
Data
Backbone
Endpoint
RabbitMQ
BBFTP
HTCondor
Satellite Processing CC-IN2P3
DRP
Satellite
Processing
Pegasus /
HTCondor
Tape
Periodic
Calibration
Payload
Template
Generation
Payload
Raw Calib
Validation
Payload
Alert
Production
Payload
Annual
Calibration
Payload
DRP
Payload
MOPS
Payload
Daily Cal.
Update
Payload
Chilean Data Access Center
Science
Platform
(DAC)
Data
Backbone
Endpoint
Science
Users
Staff
Staff
Alert
Users
Community
Alert
Brokers
EPO
Other Data
Partners
Data
Backbone
Endpoint
Figure
Data
3:
Management System Deployment
The contents of this document are subject to configuration control by the
Team.
11
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
NCSA Enclave
Base Enclave
US Data Access Center
Commissioning Cluster
Satellite Processing CC-IN2P3
Chilean Data Access Center
Workload /
Workflow
Provisioning /
Deployment
HTCondor
vSphere +
Puppet
Management /
Monitoring
Workload /
Workflow
Provisioning /
Deployment
HTCondor
Management /
Monitoring
Workload /
Workflow
Provisioning /
Deployment
HTCondor
Management /
Monitoring
Management /
Monitoring
Workload /
Workflow
Provisioning /
Deployment
HTCondor
Management /
Monitoring
Workload /
Workflow
Provisioning /
Deployment
HTCondor
Management /
Monitoring
Identity
Management
Identity
Management
Replica
Figure
Data
4:
Management System Common Infrastructure Services
The contents of this document are subject to configuration control by the
Team.
12
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
•Low-level astronomical software primitives
afw
)
and data
•Image processing and measurement framework
ip_*
,
meas_*
)with core
•Additional image processing and
meas_extensions_*
measurement
)
algorithms
•High-level algorithms and driver
pipe_tasks
scripts
,
pipe_drivers
that
) define pipelines
•Camera-specific customizations
obs_*
)
(
Figure
Data
5:
Management Science Pipelines Software “Stack”
Key middleware components include:
The contents of this document are subject to configuration control by the
Team.
13
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
•Data access client
daf_persistence
(Data Butler) (
•Parallel distributed
qserv
)
database (
•Task framework
pex_*
,
log
,
pipe_base
(
,
ctrl_pool
)
•Workflow and orchestration for
ctrl_*
production
)
control (
Infrastructure components include:
•Other databases (typically relational)
•Filesystems
•Authentication and authorization (identity management)
•Provisioning and resource management
•Monitoring
The relationships between the middleware and infrastructure
Figure
6.
5 Base Enclave
Services located in this enclave are located at the Base solely
the OCS or the Camera Data System (also known as the Camera
cases, services located here interact closely with corresponding
Level 1 Domain, to the point where the Base service cannot
operational. This reliance has been taken into account in
The primary goals of the services in this enclave are to transfer
either to NCSA, from NCSA, or to the Data Backbone.
The services in this enclave and their partners in the NCSA
to run rapidly and reliably. They run at times (outside office
The contents of this document are subject to configuration control by the
Team.
14
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Task Framework
Config API
Logging API
MultiNodeTask
API
MultiCoreTask
API
Control and
Orchestration
Authentication
& Authorization
Provisioning &
Resource
Management
Monitoring
Data Butler
Science Algorithms
SuperTask
Task
Command Line
Activator
SUIT Activator
Pre-Flight and
Job-Level
Activators
Data Backbone
RDBMS
Qserv
Files
Figure
Data
6:
Management Middleware and Infrastructure
The contents of this document are subject to configuration control by the
Team.
15
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
are not amenable to a human-in-the-loop design. Instead,
tonomously, often under the control of the OCS, with human
trol only at the highest level.
5.1 Service Descriptions
Detailed concepts of operations for each service can be found
the LSST Production
LDM-230
Services”
).
(
5.1.1 Image and EFD Archiving
This component is composed of several Image Archiving service
instances: one pair each for the LSSTCam, the ComCam, and
graph, all of which may be operated simultaneously. These
camera, including the wavefront sensors and the guide sensors
when so configured, retrieving them from their respective
They also capture specific sets of metadata associated with
values and event timings, from the OCS publish/subscribe
The image pixels and metadata are then permanently archived
catch-up versions archive into the Data Backbone any raw
missed by the primary archiving services due to network
from the flash storage in the Camera Data System instances
This component also includes an EFD Transformation service
(including telemetry, events, configurations, and commands)
annex, transforms it into a form more suitable for querying
into the permanently archived “Transformed EFD” database
5.1.2 Prompt Processing Ingest
This component is composed of two instances that capture
the LSSTCam and ComCam Camera Data Systems along with selected
and/or EFD and transfer them to the Prompt Processing service
Domain.
The contents of this document are subject to configuration control by the
Team.
16
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
There is no Prompt Processing Ingest instance for the auxiliary
5.1.3 OCS Driven Batch Control
This service receives commands from the OCS and invokes the
service in the NCSA Enclave Level 1 Domain to execute corresponding
used for modest-latency analysis of images during Commissioning
calibration images in normal observing operations. A summary
formed is returned to the OCS for each command, following
protocol.
5.1.4 Telemetry Gateway
This service obtains information from the NCSA Enclave Level
and quality metrics from Prompt Processing of images and
and transmits it to the OCS as specified in the Data Management-OCS
tion Interface
LSE-72
). Note
(
that more detailed information on the
of DMS services will also be available to Observatory operators
inated from the Management/Monitoring infrastructure
environments.
5.2 Interfaces
OCS to all Base Enclave services: these interface through
subsystem.
Archiver and Catch-Up Archiver to Data Backbone: files are
age via a file transfer mechanism, and their information
Data Backbone management dataabases. The Data Butler is
science-payload interface.
EFD to EFD Transformer: this interface is via connection
EFD as well as file transfer from the EFD’s Large File Annex.
EFD Transformer to Data Backbone: Transformed EFD entries
The contents of this document are subject to configuration control by the
Team.
17
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
formed EFD” database resident within the Data Backbone.
Camera Data System to Archiver, Catch-Up Archiver, Prompt
face through the custom library provided by the Camera Data
Prompt Processing Ingest to Prompt Processing: BBFTP is
ternational network from the ingest service to the processing
OCS Driven Batch Control to OCS Driven Batch Processing:
cution instructions over the international network from
service and return status and result information.
Telemetry Gateway from NCSA Enclave Level 1 Domain services:
status and quality metrics to the gateway over the international
6 NCSA Enclave Level 1 Domain
This domain is responsible for the compute-intensive processing
tions and other operations closely tied with the Observatory.
images and metadata from the Observatory into “online” science
them to the DACs, alert subscribers, and back to the OCS.
The Prompt Processing, OCS Driven Batch Processing, and
port execution of science payloads in three different ways.
tegrated with the observing cadence and is intended to function
result deadlines. OCS Driven Batch Processing is invoked
latency requirements. Offline Processing is not invoked by
control, typically during the daytime.
The Alert Distribution and Alert Filtering services receive
Processing of each scienc visit; they then provide alert
and LSST data rights holders, respectively.
The Level 1 Quality Control service monitors the “online”
alerts, notifying operators if any anomalies are found.
The contents of this document are subject to configuration control by the
Team.
18
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Like the services in the Base Enclave, these services need
are designed to execute autonomously.
6.1 Service Descriptions
Detailed concepts of operations for each service can be found
the LSST Production
LDM-230
Services”
).
(
6.1.1 Prompt Processing
This service receives crosstalk-corrected images and metadata
Ingest service at the Base and executes the Alert Production
erating “online” data products that are stored in the Data
payload then sends alerts to the Alert Distribution service.
The Prompt Processing service has calibration (including
science, and deep drilling modes. In calibration mode,
tion payload that provides rapid feedback of raw calibration
mode, two consecutive exposures are grouped and processed
of exposure groupings to be processed as visits in deep drilling
The service is required to deliver Alerts within 60 seconds
standard science visit with 98% reliability.
There is no Prompt Processing service instance for the Auxiliary
6.1.2 OCS Driven Batch Processing
This service executes science payloads in response to commands
Control service at the Base and thus indirectly from the
used for modest-latency analysis of images during Commissioning
calibration images in normal observing operations. Images
the Data Backbone, and results are provided back to the Data
connection from this service to the Camera Data System. This
latency from image acquisition to processing start by the
Data Backbone transfer. A summary status for the processing
The contents of this document are subject to configuration control by the
Team.
19
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Driven Batch Control service to be returned to the OCS.
6.1.3 Offline Processing
This service executes science payloads to ensure that all
within 24 hours. In particular, this service executes the
System payload. It also may execute a variant of the Alert
Processing service encounters difficulties. Images and
Backbone, and results are provided back to the Data Backbone.
6.1.4 Level 1 Quality Control
This service collects information on Level 1 science and
processes the science data products from the Data Backbone
surements, and monitors the measurement values against
automated quality control capability for potentially detecting
telescope, camera, data acquisition, or data processing.
crossings are delivered to Observatory operators and to
tists for verification, analysis, and resolution.
6.1.5 Alert Distribution
This service obtains alerts generated by the Alert Production
them to community alert brokers and to the Alert Filtering
6.1.6 Alert Filtering
This service obtains an alert feed from the Alert Broker
LSST data rights holders to execute limited filters against
then distributed to the individuals.
The contents of this document are subject to configuration control by the
Team.
20
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
6.2 Interfaces
Prompt Processing to Alert Distribution and Alert Filtering:
transport system.
Prompt Processing to Offline Processing: in the event that
lotted time window, processing can be cancelled and the failure
Processing will redo the processing at a later time. Note
computational resources have been provisioned, for the
continue to run, with spare capacity used to maintain latency
there would effectively be an infinite time window.
Science Payloads to Data Backbone: payloads use the Data
and catalog databases within the Data Backbone.
7 NCSA Enclave Level 2 Domain
This domain is responsible for all longer-period data processing
largest and most complex payloads supported by the DMS: the
tion (DRP) and periodic Calibration Products Productions
even while the annual DRP is executing, hence the need for
Quality Control Service monitors the science data products,
lies are found.
The services in this domain need to run efficiently and reliably
ning weeks or months. They need to execute millions or billions
becomes available while tracking the status of each and
signed to execute autonomously with human oversight, monitoring,
the highest level, although provisions are made for manual
sary.
This domain does not have direct users (besides the operators
within it obtain inputs from the Data Backbone and place
bone.
The contents of this document are subject to configuration control by the
Team.
21
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
7.1 Service Descriptions
7.1.1 Template and Calibration Products Production Execution
This service executes various CPP science payloads at intervals
tion Images and populate the Calibration Database with
of raw calibration images from the Data Backbone and information
This includes the computation of crosstalk correction matrices.
product, the templates used by Alert Production are also
raw science images from the Data Backbone. Additional information
logs are also taken from the Data Backbone. The intervals
depend on the stability of Observatory systems but are expected
and annual executions. The annual execution is a prerequisite
of the Data Release Production. The service involves human
termine initial configurations of the payload, to monitor
to provide additional configuration information during execution.
7.1.2 Data Release Production Execution
This service executes the DRP science payload annually to
after the annual CPP is executed. A small-scale (about
executed first to ensure readiness, followed by the full
taken from the Data Backbone along with Master Calibration
the Transformed EFD. Additional information such as external
from the Data Backbone. Computing is performed in conjunction
cessing service at CC-IN2P3, which will have capacity for
data products from both the mini-production and the main
Data Backbone, including both images and catalogs. From
staff scientists and selected external scientists using
LSST Science Platform to ensure quality and readiness for
products are loaded into the Data Access Center services,
release date. The service involves human scientist/operator/programmer
initial configurations of the payload, to monitor and analyze
necessary, to make “hot fixes” during execution that maintain
resulting data products.
The contents of this document are subject to configuration control by the
Team.
22
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
7.1.3 Level 2 Quality Control
This collects information on Level 2 science payload execution,
data products from the Data Backbone to generate additional
the measurement values against defined thresholds, providing
capability for potentially detecting issues with the data
telescope, camera, or data acquisition. Alarms stemming
ered to LSST Data Facility Production Scientists for verification,
7.2 Interfaces
Calibration Products Production Execution and Data Release
Backbone: for large-scale productions, a workflow system
lected database entries from the Data Backbone to local
payloads via the Data Butler. Similarly, the staging system
alogs into the Data Backbone.
8 Satellite Computing Enclave
8.1 Service Description
8.1.1 Data Release Production Satellite Processing
This service controls the processing of jobs on the CC-IN2P3
der the overall workload and workflow management of the Data
tion service at NCSA.
8.2 Interfaces
DRP Satellite Processing to DRP Execution: DRP Execution
cessing facility. All raw data, including images, metadata,
transferred. Intermediate data products will be transferred
Domain.
The contents of this document are subject to configuration control by the
Team.
23
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
9 Data Access Center Enclaves
There are two Data Access Centers, one in the US at NCSA
These DACs are responsible for all science-user-facing
LSST Science Platform (LSP). The LSP is the preferred analytic
in the DAC. It provides computation and data access on both
timescales. The US DAC also includes a service for distributing
nual (Data Release) timescales to partner institutions,
Public Outreach (EPO).
The services in this domain must support multiple users simultaneously
must be responsive to science user needs; updates are likely
from the other domains as a result. The LSP must operate reliably
is not impeded.
9.1 Service Descriptions
9.1.1 Bulk Data Distribution
This service is used to transmit Level 1 and Level 2 data products
ucation and Public Outreach, the UK LSST project, and the
It extracts data products from the Data Backbone and transmits
connections to designated, pre-subscribed partners.
9.1.2 LSST Science Platform DAC instances
This service provides an exploratory analysis environment
ther broken down into three “Aspects” that it presents to
“backend services” that users can take advantage
7.
of, as illustrated
The “Portal” Aspect provides a pre-specified yet flexible discovery,
“JupyterLab” Aspect provides a fully flexible (“notebook”)
ing of images, catalogs, and plots and providing for execution
algorithms. The “Web API” Aspect provides a language-independent,
vices data access API with extensions for LSST capabilities
The contents of this document are subject to configuration control by the
Team.
24
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Platform Backend Services
LSST Science Platform
LSP Web
APIs
Aspect
LSP Portal
Aspect
LSP
JupyterLab
Aspect
User
Compute
User File
Storage
User
Databases
Figure
LSST
7:
Science Platform
via all three Aspects to all data products, including images,
API Aspect regenerates “virtual” data products on demand
The backend services provide general-purpose user computation,
sion; user file storage accessible to all three Aspects; and
tables. Data may be shared with individual users, with groups,
rights holders). Resource management of the backend services
quota with additional resources allocated by a committee.
All usage of any LSST Science Platform instance requires
only to LSST data rights holders or LSST operations staff.
9.2 Interfaces
Bulk Data Distribution and LSST Science Platform to Data
vices retrieve their data, including raw images, nightly
products, metadata, and provenance, from the Data Backbone.
the LSP Web APIs to retrieve data. The LSP JupyterLab Aspect
also can use the Data Butler client library to access the
Bulk Data Distribution to partners: The exact delivery mechanism
bution is TBD.
The contents of this document are subject to configuration control by the
Team.
25
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
10 NCSA Enclave Analysis and Developer Support
This domain encompasses environments for analysts, developers,
Its users are the Observatory staff as they analyze raw data
characterize them, develop new algorithms and systems,
nents and services before deployment.
10.1 Service Descriptions
10.1.1 LSST Science Platform Science Validation instance
This instance of the LSST Science Platform is customized
intermediate data products from the Alert, Calibration
tions. It is optimized for usage by scientists within the
lected external scientists can be granted access to assist
optimization is to size and configure the three Aspects of
more JupyterLab usage and less portal usage is expected.
10.1.2 Developer Services
Software version control service, packaging, build and unit
agement, ticket tracking service, documentation services,
10.1.3 Integration and Testing
Integration environments representing various deployment
vices, test datasets, test execution services, metric measurement
This service includes the DM instance of the Camera DAQ Test
integration environment, as well as the Prototype Data Access
Access Center Enclave integration environment.
The contents of this document are subject to configuration control by the
Team.
26
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
10.2 Interfaces
LSP Science Validation instance and Integration and Testing
vices in this domain interface with the Data Backbone. The
is used to inspect, analyze, and validate the data products
prior to their release and so has access to those products
be used to annotate the data products, it can also write to
tion and Testing environment may use raw data, intermediate
products to perform tests.
Developer Services do not have direct interfaces with the
communicate via the distributed source version control
system, and the configuration system. There is a separate
Data Backbone to manage and track datasets used during development.
11 Commissioning Cluster
11.1 Service Description
11.1.1 LSST Science Platform Commissioning instance
This instance of the LSST Science Platform for Science Validation
Cluster at the Base Facility (but also has access to computational
accesses a Base endpoint for the Data Backbone. This location
of both access to Data Backbone-resident data (which does
the international network) and, perhaps more importantly,
staff in Chile, which are served locally. Note that the Commissioning
direct access to the Camera Data System; it relies on the
Commissioning Cluster will have direct access to the OCS’s
transformation).
The contents of this document are subject to configuration control by the
Team.
27
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
11.2 Interfaces
Commissioning Cluster to Data Backbone: The Commissioning
bone for its data, like the other instances of the LSST Science
Commissioning Cluster to EFD: The Commissioning Cluster
to the Base replica of the EFD (before transformation).
12 Infrastructure Domain
This domain encompasses the underlying services and systems
vironments in which the other domains are deployed and operate.
domains but has no direct users.
12.1 Service Descriptions
12.1.1 Data Backbone
The Data Backbone is a key component that provides for data
cation, allowing data products to move between computational
provides policy-based replication of files (in the Science
Science Catalog Archive) across multiple physical locations,
ing Cluster, NCSA, and DACs. It manages caches of files at
tence to long-term archival storage (e.g. tape). It provides
datasets and database entries and a retrieval mechanism
The Qserv distributed database system for large-scale catalog
Data Backbone in each DAC as well as in the NCSA Enclave.
The relationships between the Data Backbone8.
components
The contents of this document are subject to configuration control by the
Team.
28
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Data
Backbone
Endpoint
Tape
Data Butler
Client
Science Image Archive
Science Catalog Archive
Data Backbone Transport and Replication
Metadata &
Provenance
Files
Qserv
Other DBs
Caches
Science
Image
Archive
Science
Catalog
Archive
Science
Image
Archive
Science
Catalog
Archive
Data
Backbone
Endpoint
Caches
Other
Backbone
Clients
Data Butler
Client
Other
Backbone
Clients
Figure
Data
8:
Backbone
The contents of this document are subject to configuration control by the
Team.
29
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
12.1.2 Management/Monitoring
These services provide management and monitoring at service
levels for each enclave and domain.
12.1.3 Provisioning/Deployment
These services provide compute, local-to-node storage,
for all processing, including Prompt Processing, Batch
They allow allocation of compute and storage resources as
deployment of services onto those resources.
Some compute resources are reserved for particular uses,
sioned, up to a certain maximum quota, if needed to deal with
The priority order for processing is:
•Prompt processing
•Offlineprocessing
•OCS-controlled batch processing
•LSP Commissioning Cluster processing
•LSP Science Validation processing
•LSP Data Access Center processing
•Template and Calibration Products Production
•Data Release Production
The Base Enclave’s services are not highly dynamic or flexible,
terfacing to the OCS and Camera Data System. The baseline
vSphere; they will be deployed using Puppet.
The contents of this document are subject to configuration control by the
Team.
30
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
12.1.4 Workload/Workflow
These services provide management of the execution of science
gle pipeline to a series of “campaigns”, each consisting
handle massively distributed computing, executing jobs
and tracking their status and outputs. They ensure that the
to it and that outputs (including log files, if any) are preserved.
multiple computing environments, in particular between
Facility at CC-IN2P3.
12.1.5 Batch Processing
This service provides execution of batch jobs with a variety
in a variety of environments (e.g. OS and software configurations)
sioned compute resources. It will use containerization
ments. HTCondor is the baseline technology choice for this
12.1.6 Identity Management
This service provides authentication and authorization
especially the LSST Science Platform instances.
12.2 Interfaces
The infrastructure services generally interact with all
Enclave, there is no workload/workflow management or batch
deployment are also simplified.
Identity management instances are present in the Base and
will be maintained at the Summit.) These are used to support
tion for the other physically co-located enclaves: the Commissioning
Access Centers.
The contents of this document are subject to configuration control by the
Team.
31
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
13 Software Components
13.1 Science Payloads
These payloads are described in more detail inLDM-
the DM Applications
151
). Payloads are built from application software components.
13.1.1 Alert Production Payload
Executes under control of the Prompt Processing service.
products including Alerts (with the exception of Solar System
into the Data Backbone and Level 1 Database. Transmits Alerts
Generates image quality feedback to the OCS and observers
crosstalk-corrected science images and associated metadata
cessing service; uses Master Calibration Images, Template
ibration Database information from the Data Backbone.
13.1.2 MOPS Payload
Executes under control of the Offline Processing service after
plete. Generates entries in the MOPS Database and the Level
System Object records, measurements, and orbits. Performs
of transients. Uses Level 1 Database entries and images from
13.1.3 Raw Calibration Validation Payload
Executes under control of the Prompt Processing service.
quality feedback to the OCS and observers via the Telemetry
science images and associated metadata delivered by the
Calibration Images, and Calibration Database information
The contents of this document are subject to configuration control by the
Team.
32
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
13.1.4 Daily Calibration Products Update Payload
Executes under control of the OCS-controlled batch processing
can be synchronized with the observing schedule. Uses raw
mation from the Transformed EFD to generate a subset of
Calibration Database entries in the Data Backbone.
13.1.5 Periodic Calibration Products Production Payload
Executes under control of the Template and CPP Execution
tervals but perhaps as frequently as weekly or as infrequently
stability of Observatory systems and their calibrations.
formation from the Transformed EFD to generate a subset of
Calibration Database entries in the Data Backbone.
13.1.6 Template Generation Payload
Executes under control of the Template and CPP Execution
templates for Alert Production in between annual Data Release
images to generate the templates, placing them in the Data
13.1.7 Annual Calibration Products Production Payload
Executes under control of the Template and CPP Execution
to the start of the Data Release Production. Uses raw calibration
the Transformed EFD, information from the Auxiliary Telescope
catalogs to generate Master Calibration Images and Calibration
Backbone.
13.1.8 Data Release Production Payload
Executes under control of the DRP Execution service at annual
DRP” over a small portion of the sky, followed by the full
science data products in the Data Backbone.
The contents of this document are subject to configuration control by the
Team.
33
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
13.2 SUIT
The Science User Interface and Tools provide visualization,
ing, and searching elements that can be assembled into predetermined
be used flexibly within dynamic “notebook” environments.
13.3 Middleware
13.3.1 Data Butler Access Client
The Data Butler provides an access abstraction for all science
underlying data sources and destinations to be configured
ends ranging from local disk to network locations and a variety
YAML and FITS files (extensible to HDF5 or ASDF) to database
available within the LSST Science Platform JupyterLab environment.
13.3.2 Parallel Distributed Database (Qserv)
Underlying the catalog data access web service is a parallel
handle the petabyte-scale, tens-of-trillions-of-rows
13.3.3 Task Framework
The Task Framework is a Python class library that provides
entry points and conventions) to organize low-level algorithms
rithmic components (Tasks; e.g. dark frame subtraction,
ment), and to organize tasks into basic pipelines (SuperTasks;
a coadd, difference a visit). The algorithmic code is written
classes and providing implementation for standard entry
the pipelines to be constructed and run at the level of a
synchronized nodes. It allows for sub-node parallelization:
cution, as well as providing (in the future) parallelization
core Tasks and synchronized multi-node Tasks.
The Task Framework serves as an interface layer between orchestration
The contents of this document are subject to configuration control by the
Team.
34
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
code. It exposes a standard interface to “activators” (command-line
orchestration layer and QA systems), which use it to execute
Task Framework does not concern itself with fault-tolerant
pipelines over multiple (thousands) of nodes nor any staging
this is the concern of the orchestration middleware.
The Task Framework exposes to the orchestration system needs
derlying algorithmic code (i.e., the number of cores needed,
pected need for data). It may also receive from the orchestration
how to optimally run the particular task (i.e., which level
desired).
It also includes a configuration API and a logging API.
14 Design and Implementation Standards
Standards have been adopted by the DM Change Control Board
nent designs within the LSST DM System. Coding standards
to design may be found in the LSST
1]).
DM Developer Guide ([
14.1 Python Version
All LSST DM code will run under Python 3 by 2018-01-31. Packa
have their code backward-compatible with Python 2.7, but
Pipelines code that is
lsst_apps
dependency
, and then only
of
until external dependencies
become Python 3-only.
Python 2.7 will be deprecated by the time LSST begins operations,
dencies are in the process of moving to being Python 3-only.
14.2 HTTPS Protocol
In the absence of a specific technical justification and acceptance
Security Officer and DM Change Control Board, all Web-enabled
The contents of this document are subject to configuration control by the
Team.
35
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
vices exposed to users and the public Internet will use the
protocol. To reiterate: this is only a default, and exceptions
The covered interfaces include those of the three LSP Aspects
APIs).
The requirement to implement data access policies limiting
holders will require all, or nearly all, data access to be
nical justification. In addition, it appears to be appropriate
current Internet environment, in the absence of good reasons
The contents of this document are subject to configuration control by the
Team.
36
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
15 Appendix: Traceability
15.1 Requirement to Component Traceability
Note that DMS-REQ-0006 has no components; this requirement
dropped.
Requirement
Components
DMS-REQ-0059 Bad Pixel Map
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0060 Bias Residual
OCSImage
Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0061 Crosstalk Correction Matrix
OCS Driven Batch, Calibration
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0062
Illumination
Correction
Frame
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0063 Monochromatic Flatfield Data
Cube
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0130 Calibration Data Products
OCS Driven Batch, Calibration
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0132 Calibration Image Prove-
nance
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0282 Dark Current Correction
Frame
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0283 Fringe Correction
OCS Driven
Frame
Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0018 Raw Science Image Data Ac-
quisition
Image and EFD Archiving
The contents of this document are subject to configuration control by the
Team.
37
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0020 Wavefront Sensor Data Ac-
quisition
Image and EFD Archiving
DMS-REQ-0022 Crosstalk Corrected Science
Image Data Acquisition
Prompt Processing
DMS-REQ-0024 Raw Image Assembly
Image and EFD Archiving
DMS-REQ-0068 Raw Science Image Metadata Image and EFD Archiving
DMS-REQ-0265 Guider Calibration Data Ac-
quisition
Image and EFD Archiving, OCS Driven Batch,
ucts Production Execution, Annual Calibration,
tion Update, Periodic Calibration
DMS-REQ-0326 Storing Approximations of
Per-pixel Metadata
Data Release Production
DMS-REQ-0331 Computing Derived Quanti-
ties
LSP Web APIs, Data Release Production
DMS-REQ-0332 Denormalizing Database Ta-
bles
LSP Web APIs
DMS-REQ-0333 Maximum Likelihood Values
and Covariances
Data Access Client, Alert Production, Data
DMS-REQ-0346 Data Availability
Image and EFD Archiving, OCS Driven Batch,
ing, Prompt Processing, Calibration Products
cution, Data Release Production Execution
DMS-REQ-0347 Measurements in catalogs
Data Access Client, LSP Web
lease Production
DMS-REQ-0004 Nightly Data Accessible
Within 24 hrs
Alert Distribution, Alert Filtering, Offline
Processing, Alert Production, MOPS and Forced
DMS-REQ-0010 Difference Exposures
Alert Production
DMS-REQ-0074 Difference Exposure At-
tributes
Alert Production
DMS-REQ-0069 Processed Visit
AlertImages
Production
DMS-REQ-0029 Generate Photometric Zero-
point for Visit Image
Alert Production
DMS-REQ-0030 Generate WCS for Visit Im-
ages
Alert Production
DMS-REQ-0070 Generate PSF for Visit Images Alert Production
DMS-REQ-0072 Processed Visit Image Con-
tent
Alert Production
DMS-REQ-0327 Background Model Calcula-
tion
Alert Production
DMS-REQ-0328 Documenting Image Charac-
terization
Alert Production
The contents of this document are subject to configuration control by the
Team.
38
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0097 Level 1 Data Quality Report
Definition
QC System, Alert Production
DMS-REQ-0099 Level 1 Performance Report
Definition
Prompt Processing, QC System
DMS-REQ-0101 Level 1 Calibration Report
Definition
OCS Driven Batch, Daily Calibration Update,
DMS-REQ-0266 Exposure Catalog
Alert Production
DMS-REQ-0269 DIASource Catalog
Alert Production
DMS-REQ-0270 Faint DIASource Measure-
ments
Alert Production
DMS-REQ-0271 DIAObject Catalog
Alert Production
DMS-REQ-0272 DIAObject Attributes
Alert Production
DMS-REQ-0273 SSObject Catalog
MOPS and Forced Photometry
DMS-REQ-0274 Alert Content
Alert Production
DMS-REQ-0317 DIAForcedSource
Alert Production,
Catalog
MOPS and Forced Photometry
DMS-REQ-0319 Characterizing
AlertVariability
Production, MOPS and Forced Photometry
DMS-REQ-0323 Calculating SSObject Parame-
ters
LSP Web APIs
DMS-REQ-0324 Matching DIASources to Ob-
jects
LSP Web APIs, Alert Production
DMS-REQ-0325 Regenerating L1 Data Prod-
ucts During Data Release Processing
Data Release Production
DMS-REQ-0034 Associate Sources to Objects Data Release Production
DMS-REQ-0047 Provide PSF for Coadded Im-
ages
Data Release Production
DMS-REQ-0103 Produce Images
Datafor
Release
EPO
Production
DMS-REQ-0106 Coadded Image Provenance
Data Release Production
DMS-REQ-0267 Source Catalog
Data Release Production
DMS-REQ-0268 Forced-Source
DataCatalog
Release Production
DMS-REQ-0275 Object Catalog
Data Release Production
DMS-REQ-0046 Provide Photometric Red-
shifts of Galaxies
Data Release Production
DMS-REQ-0276 Object Characterization
Data Release Production
DMS-REQ-0277 Coadd Source
Data
Catalog
Release Production
DMS-REQ-0349 Detecting extended low sur-
face brightness objects
Data Release Production
DMS-REQ-0278 Coadd Image Method Con-
straints
Data Release Production
The contents of this document are subject to configuration control by the
Team.
39
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0279 Deep Detection
DataCoadds
Release Production
DMS-REQ-0280 Template Coadds
Data Release Production, Template Generation
DMS-REQ-0281 Multi-bandData
Coadds
Release Production
DMS-REQ-0329 All-Sky Visualization of Data
Releases
Data Release Production
DMS-REQ-0330 Best SeeingData
Coadds
Release Production
DMS-REQ-0334 Persisting DataBackbone,
Products LSP Web APIs
DMS-REQ-0335 PSF-Matched
Data
Coadds
Release Production
DMS-REQ-0336 Regenerating Data Products
from Previous Data Releases
Data Backbone, LSP Web APIs
DMS-REQ-0337 Detecting faint variable ob-
jects
Data Release Production
DMS-REQ-0338 Targeted Coadds
Data Backbone, LSP Web APIs
DMS-REQ-0339 Tracking Characterization
Changes Between Data Releases
Data Backbone, LSP Web APIs
DMS-REQ-0320 Processing of Data From Spe-
cial Programs
Task Execution Framework, Data Release Production
DMS-REQ-0321 Level 1 Processing of Special
Programs Data
OfflineProcessing,PromptProcessing,Alert
and Forced Photometry
DMS-REQ-0322 Special Programs Database
Data Backbone, LSP Web APIs
DMS-REQ-0344 Constraints on Level 1 Special
Program Products Generation
Offline Processing, Prompt Processing, Data
Web APIs, Alert Production, MOPS and Forced
DMS-REQ-0185 Archive Center
Bulk Distribution, Data Backbone, ITC Environments
DMS-REQ-0186 Archive Center Disaster Re-
covery
Bulk Distribution, Data Backbone, ITC Environments
DMS-REQ-0187 Archive Center Co-Location
with Existing Facility
ITC Environments
DMS-REQ-0188 Archive to Data Access Center
Network
Networks
DMS-REQ-0189 Archive to Data Access Center
Network Availability
Networks
DMS-REQ-0190 Archive to Data Access Center
Network Reliability
Networks
DMS-REQ-0191 Archive to Data Access Center
Network Secondary Link
Networks
DMS-REQ-0176 Base Facility Infrastructure
Data Backbone, ITC Environments
DMS-REQ-0177 Base Facility Temporary Stor-
age
Data Backbone
The contents of this document are subject to configuration control by the
Team.
40
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0178 Base Facility Co-Location with
Existing Facility
ITC Environments
DMS-REQ-0316 Commissioning
ITC Environments
Cluster
DMS-REQ-0180 Base to Archive
Networks
DMS-REQ-0181 Base to Archive Network
Availability
Networks
DMS-REQ-0182 Base to Archive Network Reli-
ability
Networks
DMS-REQ-0183 Base to Archive Network Sec-
ondary Link
Networks
DMS-REQ-0008 Pipeline Availability
Alert Distribution, Alert Filtering, Image
Driven Batch, Offline Processing, Prompt Processing,
try Gateway, Calibration Products Production
Release Production Execution, Data Backbone,
ments, Networks
DMS-REQ-0161 Optimization of Cost, Reliabil-
ity and Availability in Order
Alert Distribution, Alert Filtering, Data
ments, Networks, LSP Web APIs, LSP JupyterLab,
DMS-REQ-0162 Pipeline Throughput
Alert Distribution, Alert Filtering, Image
OCS Driven Batch, Offline Processing, Prompt
Backbone, ITC Environments, Networks
DMS-REQ-0163 Re-processing
Calibration
CapacityProducts Production Execution,
duction Execution, Data Backbone, ITC Environments,
works
DMS-REQ-0164 Temporary Storage for Com-
munications Links
Data Backbone
DMS-REQ-0165 Infrastructure Sizing for
“catching up”
Image and EFD Archiving, Offline Processing,
ITC Environments, Networks
DMS-REQ-0166 Incorporate Fault-Tolerance
Data Backbone
DMS-REQ-0167 Incorporate
Alert
Autonomics
Distribution, Alert Filtering, Image
Driven Batch, Offline Processing, Prompt Processing,
tion Products Production Execution, Data
Execution, Data Backbone, ITC Environments,
DMS-REQ-0314 Compute Platform Hetero-
geneity
Alert Distribution, Alert Filtering, Image
OCS Driven Batch, Offline Processing, Prompt
Telemetry Gateway, Calibration Products
tion, Data Release Production Execution,
Proposal Manager, Developer Services, Integration
Data Backbone, Identity Manager, ITC Environments,
works, Data Access Client, Task Execution Framework,
APIs, LSP JupyterLab, LSP Portal, QC System
The contents of this document are subject to configuration control by the
Team.
41
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0318 Data Management Unsched-
uled Downtime
Alert Distribution, Alert Filtering, Image
OCS Driven Batch, Offline Processing, Prompt
Telemetry Gateway, Calibration Products
tion, Data Release Production Execution,
Proposal Manager, Data Backbone, Identity
ronments, Networks, LSP Web APIs, LSP JupyterLab,
QC System
DMS-REQ-0122 Access to catalogs for exter-
nal Level 3 processing
Bulk Distribution, Data Backbone
DMS-REQ-0123 Access to input catalogs for
DAC-based Level 3 processing
Data Backbone, LSP Web APIs, LSP JupyterLab,
DMS-REQ-0124 Federation with external cat-
alogs
Data Backbone, LSP Web APIs, LSP JupyterLab,
DMS-REQ-0126 Access to images for external
Level 3 processing
Bulk Distribution, Data Backbone
DMS-REQ-0127 Access to input images for
DAC-based Level 3 processing
Data Backbone, LSP Web APIs, LSP JupyterLab,
DMS-REQ-0193 Data AccessBulk
Centers
Distribution, Data Backbone, ITC Environments,
works, LSP Web APIs, LSP JupyterLab, LSP Portal
DMS-REQ-0194 Data Access Center Simulta-
neous Connections
Networks, LSP JupyterLab, LSP Portal
DMS-REQ-0196 Data Access Center Geo-
graphical Distribution
ITC Environments
DMS-REQ-0197 No Limit on Data Access Cen-
ters
Data Backbone, Identity Manager, ITC Environments,
works, LSP Web APIs, LSP JupyterLab, LSP Portal
DMS-REQ-0075 Catalog Queries
LSP Web APIs
DMS-REQ-0077 Maintain Archive Publicly Ac-
cessible
Data Backbone
DMS-REQ-0078 Catalog Export
LSP Web
Formats
APIs
DMS-REQ-0094 Keep Historical Alert Archive
Data Backbone
DMS-REQ-0102 Provide Engineering and Fa-
cility Database Archive
Image and EFD Archiving, Data Backbone
DMS-REQ-0309 Raw Data Archiving Reliability Image and EFD Archiving,
DMS-REQ-0310 Un-Archived Data Product
Cache
Data Backbone
DMS-REQ-0311Regenerate Un-archived Data
Products
Data Backbone, LSP Web APIs
DMS-REQ-0312 Level 1 Data Product Access
Prompt Processing, Data
duction
The contents of this document are subject to configuration control by the
Team.
42
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0313 Level 1 and 2 Catalog Access
Data Backbone, LSP Web APIs
DMS-REQ-0341 Providing a Precovery Service Offline Processing, LSP
DMS-REQ-0345 Logging of catalog queries
LSP Web APIs
DMS-REQ-0168 Summit Facility Data Commu-
nications
Networks
DMS-REQ-0170 Prefer Computing and Stor-
age Down
ITC Environments
DMS-REQ-0315 DMS Communication with
OCS
Image and EFD Archiving, OCS Driven Batch,
ing, Telemetry Gateway
DMS-REQ-0171 Summit to Base
Networks
DMS-REQ-0172 Summit to Base Network
Availability
Networks
DMS-REQ-0173 Summit to Base Network Re-
liability
Networks
DMS-REQ-0174 Summit to Base Network Sec-
ondary Link
Networks
DMS-REQ-0175 Summit to Base Network
Ownership and Operation
Networks
DMS-REQ-0002 Transient Alert Distribution
Alert Distribution, Alert
duction
DMS-REQ-0089 Solar System Objects Avail-
able Within Specified Time
Offline Processing, Data Backbone, LSP Web APIs,
Forced Photometry
DMS-REQ-0096 Generate Data Quality Report
Within Specified Time
QC System
DMS-REQ-0098 Generate DMS Performance
Report Within Specified Time
QC System
DMS-REQ-0100 Generate Calibration Report
Within Specified Time
QC System, Daily Calibration Update
DMS-REQ-0131 Calibration Images Available
Within Specified Time
OCS Driven Batch, Prompt Processing, Daily
date, Raw Calibration
DMS-REQ-0284 Level-1 Production Complete-
ness
Image and EFD Archiving, Offline Processing,
ing
DMS-REQ-0285 Level 1 Source Association
Alert Production
DMS-REQ-0286 SSObject Precovery
MOPS and Forced Photometry
DMS-REQ-0287 DIASource Precovery
Data Backbone, LSP Web APIs, MOPS and Forced
DMS-REQ-0288UseofExternalOrbitCatalogs Alert Production, MOPS
DMS-REQ-0342 Alert Filtering
AlertService
Filtering
DMS-REQ-0343 Performance Requirements
for LSST Alert Filtering Service
Alert Distribution, Alert Filtering
The contents of this document are subject to configuration control by the
Team.
43
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0348 Pre-definedAlert
alertFiltering
filters
DMS-REQ-0289 Calibration Production Pro-
cessing
OCS Driven Batch, Calibration Products Production
Annual Calibration, Daily Calibration Update,
tion
DMS-REQ-0006 Timely Publication of Level 2
Data Releases
DMS-REQ-0350 Associating Objects across
data releases
Data Backbone, Data Release Production
DMS-REQ-0291 Query Repeatibility
Data Backbone, LSP Web APIs
DMS-REQ-0292 Uniqueness of IDs Across
Data Releases
Data Backbone, LSP Web APIs
DMS-REQ-0293 Selection of Datasets
Backbone, LSP Web APIs
DMS-REQ-0294 ProcessingOCS
of Datasets
Driven Batch, Offline Processing, Prompt
ibration Products Production Execution, Data
tion Execution, Data Backbone
DMS-REQ-0295 Transparent
LSP
Data
WebAccess
APIs
DMS-REQ-0119 DAC resource allocation for
Level 3 processing
Bulk Distribution, Proposal Manager, LSP
JupyterLab, LSP Portal
DMS-REQ-0120 Level 3 Data Product Self Con-
sistency
Data Backbone, Workload+Workflow, LSP Web APIs
DMS-REQ-0121 Provenance for Level 3 pro-
cessing at DACs
Data Backbone, Workload+Workflow, Data Access
Execution Framework, LSP Web APIs
DMS-REQ-0125 Software framework for Level
3 catalog processing
Data Backbone, Workload+Workflow, Data Access
Execution Framework, LSP Web APIs
DMS-REQ-0128 Software framework for Level
3 image processing
Data Backbone, Workload+Workflow, Data Access
Execution Framework, LSP Web APIs
DMS-REQ-0290 Level 3
Data
Import
Backbone, LSP Web APIs
DMS-REQ-0340 Access Controls of Level 3
Data Products
Data Backbone, Identity Manager
DMS-REQ-0009 Simulated Data
Alert Production, Annual Calibration, Daily
Data Release Production, MOPS and Forced Photometry,
odic Calibration, Raw Calibration, Template
DMS-REQ-0032 Image Differencing
Science Algorithms, Alert Production, Data
DMS-REQ-0033 Provide Source Detection
Software
Science Algorithms, Alert Production, Data
DMS-REQ-0042 Provide Astrometric Model
ScienceAlgorithms,Alert
Science Primitives
DMS-REQ-0043 Provide Calibrated Photome-
try
Science Algorithms, Alert Production, Data
Science Primitives
The contents of this document are subject to configuration control by the
Team.
44
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Requirement
Components
DMS-REQ-0052 Enable a Range of Shape
Measurement Approaches
Science Algorithms, Alert Production, Data
Science Primitives
DMS-REQ-0160 Provide User Interface Ser-
vices
LSP Portal
DMS-REQ-0296 Pre-cursor,
Data
andBackbone,
Real Data Data Access Client, Science
DMS-REQ-0351 Provide Beam Projector Coor-
dinate Calculation Software
Science Primitives
DMS-REQ-0308 Software Architecture to En-
able Community Re-Use
Science Algorithms, Alert Production, Annual
Calibration Update, Data Release Production,
Forced Photometry, Periodic Calibration,
Template Generation
DMS-REQ-0065 Provide Image Access Ser-
vices
LSP Web APIs
DMS-REQ-0155 Provide Data Access Services LSP Web APIs
DMS-REQ-0298 Data Product and Raw Data
Access
Data Backbone, LSP Web APIs
DMS-REQ-0299 Data Product
Data
Ingest
Backbone, LSP Web APIs
DMS-REQ-0300 Bulk Download
Bulk
Service
Distribution, Data Backbone
DMS-REQ-0156 Provide Pipeline Execution
Services
Workload+Workflow
DMS-REQ-0302 ProductionWorkload+Workflow
Orchestration
DMS-REQ-0303 ProductionWorkload+Workflow
Monitoring
DMS-REQ-0304 Production Fault Tolerance
Workload+Workflow, Task
DMS-REQ-0158 Provide Pipeline Construction
Services
Task Execution Framework
DMS-REQ-0305 Task Specification
Task Execution Framework
DMS-REQ-0306 Task Configuration
Task Execution Framework
DMS-REQ-0297 DMS Initialization Compo-
nent
ITC Environments
DMS-REQ-0301 Control of Level-1 Production Prompt Processing
DMS-REQ-0307 Unique Processing Coverage Workload+Workflow
15.2 Component to Requirement Traceability
Note that only “leaf” components are traced to requirements.
Component
Requirements
Data Management
The contents of this document are subject to configuration control by the
Team.
45
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Level 1 System
Alert Distribution
•DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0161 Optimization of Cost, Reliability and
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0002 Transient Alert Distribution
•DMS-REQ-0343 Performance Requirements for LSST
vice
Alert Filtering
•DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0161 Optimization of Cost, Reliability and
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0002 Transient Alert Distribution
•DMS-REQ-0342 Alert Filtering Service
•DMS-REQ-0343 Performance Requirements for LSST
vice
•DMS-REQ-0348 Pre-defined alert filters
Image and EFD Archiving
•DMS-REQ-0018 Raw Science Image Data Acquisition
•DMS-REQ-0020 Wavefront Sensor Data Acquisition
•DMS-REQ-0024 Raw Image Assembly
•DMS-REQ-0068 Raw Science Image Metadata
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0346 Data Availability
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0165 Infrastructure Sizing for “catching
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0102 Provide Engineering and Facility Database
•DMS-REQ-0309 Raw Data Archiving Reliability
•DMS-REQ-0315 DMS Communication with OCS
•DMS-REQ-0284 Level-1 Production Completeness
The contents of this document are subject to configuration control by the
Team.
46
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
OCS Driven Batch
•DMS-REQ-0059 Bad Pixel Map
•DMS-REQ-0060 Bias Residual Image
•DMS-REQ-0061 Crosstalk Correction Matrix
•DMS-REQ-0062 Illumination Correction Frame
•DMS-REQ-0063 Monochromatic Flatfield Data Cube
•DMS-REQ-0130 Calibration Data Products
•DMS-REQ-0132 Calibration Image Provenance
•DMS-REQ-0282 Dark Current Correction Frame
•DMS-REQ-0283 Fringe Correction Frame
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0346 Data Availability
•DMS-REQ-0101 Level 1 Calibration Report Definition
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0315 DMS Communication with OCS
•DMS-REQ-0131 Calibration Images Available Within Specified
•DMS-REQ-0289 Calibration Production Processing
•DMS-REQ-0294 Processing of Datasets
The contents of this document are subject to configuration control by the
Team.
47
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Offline Processing
•DMS-REQ-0346 Data Availability
•DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
•DMS-REQ-0321 Level 1 Processing of Special Programs
•DMS-REQ-0344 Constraints on Level 1 Special Program
ation
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0165 Infrastructure Sizing for “catching up”
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0341 Providing a Precovery Service
•DMS-REQ-0089 Solar System Objects Available Within
•DMS-REQ-0284 Level-1 Production Completeness
•DMS-REQ-0294 Processing of Datasets
Prompt Processing
•DMS-REQ-0022 Crosstalk Corrected Science Image Data
•DMS-REQ-0346 Data Availability
•DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
•DMS-REQ-0099 Level 1 Performance Report Definition
•DMS-REQ-0321 Level 1 Processing of Special Programs
•DMS-REQ-0344 Constraints on Level 1 Special Program
ation
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0312 Level 1 Data Product Access
•DMS-REQ-0315 DMS Communication with OCS
•DMS-REQ-0002 Transient Alert Distribution
•DMS-REQ-0131 Calibration Images Available Within Specified
•DMS-REQ-0284 Level-1 Production Completeness
•DMS-REQ-0294 Processing of Datasets
•DMS-REQ-0301 Control of Level-1 Production
The contents of this document are subject to configuration control by the
Team.
48
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Telemetry Gateway
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled
•DMS-REQ-0315 DMS Communication with OCS
Level 2 System
Calibration Products Production Execution
•DMS-REQ-0059 Bad Pixel Map
•DMS-REQ-0060 Bias Residual Image
•DMS-REQ-0061 Crosstalk Correction Matrix
•DMS-REQ-0062 Illumination Correction Frame
•DMS-REQ-0063 Monochromatic Flatfield Data
•DMS-REQ-0130 Calibration Data Products
•DMS-REQ-0132 Calibration Image Provenance
•DMS-REQ-0282 Dark Current Correction Frame
•DMS-REQ-0283 Fringe Correction Frame
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0346 Data Availability
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0163 Re-processing Capacity
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled
•DMS-REQ-0289 Calibration Production Processing
•DMS-REQ-0294 Processing of Datasets
The contents of this document are subject to configuration control by the
Team.
49
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Data Release Production Execution
•DMS-REQ-0346 Data Availability
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0163 Re-processing Capacity
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0294 Processing of Datasets
Data Access Center
Bulk Distribution
•DMS-REQ-0185 Archive Center
•DMS-REQ-0186 Archive Center Disaster Recovery
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0122 Access to catalogs for external
•DMS-REQ-0126 Access to images for external Level
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0119 DAC resource allocation for Level
•DMS-REQ-0300 Bulk Download Service
Proposal Manager
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0119 DAC resource allocation for Level
Analysis and Developer Services
Developer Services
•DMS-REQ-0314 Compute Platform Heterogeneity
Integration and Test
•DMS-REQ-0314 Compute Platform Heterogeneity
Infrastructure
Data Backbone
•DMS-REQ-0334 Persisting Data Products
•DMS-REQ-0336RegeneratingDataProductsfrom
•DMS-REQ-0338 Targeted Coadds
•DMS-REQ-0339 Tracking Characterization Changes
leases
•DMS-REQ-0322 Special Programs Database
•DMS-REQ-0344 Constraints on Level 1 Special
ation
•DMS-REQ-0185 Archive Center
•DMS-REQ-0186 Archive Center Disaster Recovery
•DMS-REQ-0176 Base Facility Infrastructure
•DMS-REQ-0177 Base Facility Temporary Storage
The contents of this document are subject to configuration control by the
Team.
50
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Data Backbone (continued)
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0161 Optimization of Cost, Reliability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0163 Re-processing Capacity
•DMS-REQ-0164 Temporary Storage for Communications
•DMS-REQ-0165 Infrastructure Sizing for “catching
•DMS-REQ-0166 Incorporate Fault-Tolerance
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0122 Access to catalogs for external Level
•DMS-REQ-0123 Access to input catalogs for DAC-based
•DMS-REQ-0124 Federation with external catalogs
•DMS-REQ-0126 Access to images for external Level
•DMS-REQ-0127 Access to input images for DAC-based
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0077 Maintain Archive Publicly Accessible
•DMS-REQ-0094 Keep Historical Alert Archive
•DMS-REQ-0102 Provide Engineering and Facility Database
•DMS-REQ-0309 Raw Data Archiving Reliability
•DMS-REQ-0310 Un-Archived Data Product Cache
•DMS-REQ-0311 Regenerate Un-archived Data Products
•DMS-REQ-0312 Level 1 Data Product Access
•DMS-REQ-0313 Level 1 and 2 Catalog Access
•DMS-REQ-0089 Solar System Objects Available Within
•DMS-REQ-0287 DIASource Precovery
•DMS-REQ-0350 Associating Objects across data releases
•DMS-REQ-0291 Query Repeatibility
•DMS-REQ-0292 Uniqueness of IDs Across Data Releases
•DMS-REQ-0293 Selection of Datasets
•DMS-REQ-0294 Processing of Datasets
•DMS-REQ-0120 Level 3 Data Product Self Consistency
•DMS-REQ-0121 Provenance for Level 3 processing at
•DMS-REQ-0125 Software framework for Level 3 catalog
•DMS-REQ-0128 Software framework for Level 3 image
•DMS-REQ-0290 Level 3 Data Import
•DMS-REQ-0340 Access Controls of Level 3 Data Products
•DMS-REQ-0296 Pre-cursor, and Real Data
•DMS-REQ-0298 Data Product and Raw Data Access
The contents of this document are subject to configuration control by the
Team.
51
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Data Backbone (continued)
•DMS-REQ-0299 Data Product Ingest
•DMS-REQ-0300 Bulk Download Service
Identity Manager
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0340 Access Controls of Level 3 Data Products
ITC Environments
•DMS-REQ-0185 Archive Center
•DMS-REQ-0186 Archive Center Disaster Recovery
•DMS-REQ-0187 Archive Center Co-Location with Existing
•DMS-REQ-0176 Base Facility Infrastructure
•DMS-REQ-0178 Base Facility Co-Location with Existing
•DMS-REQ-0316 Commissioning Cluster
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0161 Optimization of Cost, Reliability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0163 Re-processing Capacity
•DMS-REQ-0165 Infrastructure Sizing for “catching
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0196 Data Access Center Geographical Distribution
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0170 Prefer Computing and Storage Down
•DMS-REQ-0297 DMS Initialization Component
The contents of this document are subject to configuration control by the
Team.
52
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Networks
•DMS-REQ-0188 Archive to Data Access Center Network
•DMS-REQ-0189 Archive to Data Access Center Network Availability
•DMS-REQ-0190 Archive to Data Access Center Network Reliability
•DMS-REQ-0191 Archive to Data Access Center Network Secondary
•DMS-REQ-0180 Base to Archive Network
•DMS-REQ-0181 Base to Archive Network Availability
•DMS-REQ-0182 Base to Archive Network Reliability
•DMS-REQ-0183 Base to Archive Network Secondary Link
•DMS-REQ-0008 Pipeline Availability
•DMS-REQ-0161 Optimization of Cost, Reliability and Availability
•DMS-REQ-0162 Pipeline Throughput
•DMS-REQ-0163 Re-processing Capacity
•DMS-REQ-0165 Infrastructure Sizing for “catching up”
•DMS-REQ-0167 Incorporate Autonomics
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0194 Data Access Center Simultaneous Connections
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0168 Summit Facility Data Communications
•DMS-REQ-0171 Summit to Base Network
•DMS-REQ-0172 Summit to Base Network Availability
•DMS-REQ-0173 Summit to Base Network Reliability
•DMS-REQ-0174 Summit to Base Network Secondary Link
•DMS-REQ-0175 Summit to Base Network Ownership and Operation
The contents of this document are subject to configuration control by the
Team.
53
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Workload+Workflow
•DMS-REQ-0120 Level 3 Data Product Self Consistency
•DMS-REQ-0121 Provenance for Level 3 processing at
•DMS-REQ-0125 Software framework for Level 3 catalog
•DMS-REQ-0128 Software framework for Level 3 image
•DMS-REQ-0156 Provide Pipeline Execution Services
•DMS-REQ-0302 Production Orchestration
•DMS-REQ-0303 Production Monitoring
•DMS-REQ-0304 Production Fault Tolerance
Data Access Client
•DMS-REQ-0333 Maximum Likelihood Values and Covariances
•DMS-REQ-0347 Measurements in catalogs
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0121 Provenance for Level 3 processing at
•DMS-REQ-0125 Software framework for Level 3 catalog
•DMS-REQ-0128 Software framework for Level 3 image
•DMS-REQ-0296 Pre-cursor, and Real Data
Task Execution Framework
•DMS-REQ-0320 Processing of Data From Special Programs
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0121 Provenance for Level 3 processing at
•DMS-REQ-0125 Software framework for Level 3 catalog
•DMS-REQ-0128 Software framework for Level 3 image
•DMS-REQ-0304 Production Fault Tolerance
•DMS-REQ-0158 Provide Pipeline Construction Services
•DMS-REQ-0305 Task Specification
•DMS-REQ-0306 Task Configuration
Science Platform
LSP Web APIs
•DMS-REQ-0331 Computing Derived Quantities
•DMS-REQ-0332 Denormalizing Database Tables
•DMS-REQ-0347 Measurements in catalogs
•DMS-REQ-0323 Calculating SSObject Parameters
•DMS-REQ-0324 Matching DIASources to Objects
•DMS-REQ-0334 Persisting Data Products
•DMS-REQ-0336RegeneratingDataProductsfromPrevious
•DMS-REQ-0338 Targeted Coadds
•DMS-REQ-0339 Tracking Characterization Changes
leases
•DMS-REQ-0322 Special Programs Database
•DMS-REQ-0344 Constraints on Level 1 Special Program
ation
•DMS-REQ-0161 Optimization of Cost, Reliability
The contents of this document are subject to configuration control by the
Team.
54
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
LSP Web APIs (continued)
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0123 Access to input catalogs for DAC-based
•DMS-REQ-0124 Federation with external catalogs
•DMS-REQ-0127 Access to input images for DAC-based
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0075 Catalog Queries
•DMS-REQ-0078 Catalog Export Formats
•DMS-REQ-0311 Regenerate Un-archived Data Products
•DMS-REQ-0312 Level 1 Data Product Access
•DMS-REQ-0313 Level 1 and 2 Catalog Access
•DMS-REQ-0345 Logging of catalog queries
•DMS-REQ-0089 Solar System Objects Available Within
•DMS-REQ-0287 DIASource Precovery
•DMS-REQ-0291 Query Repeatibility
•DMS-REQ-0292 Uniqueness of IDs Across Data Releases
•DMS-REQ-0293 Selection of Datasets
•DMS-REQ-0295 Transparent Data Access
•DMS-REQ-0119 DAC resource allocation for Level 3
•DMS-REQ-0120 Level 3 Data Product Self Consistency
•DMS-REQ-0121 Provenance for Level 3 processing at
•DMS-REQ-0125 Software framework for Level 3 catalog
•DMS-REQ-0128 Software framework for Level 3 image
•DMS-REQ-0290 Level 3 Data Import
•DMS-REQ-0065 Provide Image Access Services
•DMS-REQ-0155 Provide Data Access Services
•DMS-REQ-0298 Data Product and Raw Data Access
•DMS-REQ-0299 Data Product Ingest
The contents of this document are subject to configuration control by the
Team.
55
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
LSP JupyterLab
•DMS-REQ-0161 Optimization of Cost, Reliability and Availability
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0123 Access to input catalogs for DAC-based Level
•DMS-REQ-0124 Federation with external catalogs
•DMS-REQ-0127 Access to input images for DAC-based Level
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0194 Data Access Center Simultaneous Connections
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0119 DAC resource allocation for Level 3 processing
LSP Portal
•DMS-REQ-0161 Optimization of Cost, Reliability and Availability
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0123 Access to input catalogs for DAC-based Level
•DMS-REQ-0124 Federation with external catalogs
•DMS-REQ-0127 Access to input images for DAC-based Level
•DMS-REQ-0193 Data Access Centers
•DMS-REQ-0194 Data Access Center Simultaneous Connections
•DMS-REQ-0197 No Limit on Data Access Centers
•DMS-REQ-0341 Providing a Precovery Service
•DMS-REQ-0119 DAC resource allocation for Level 3 processing
•DMS-REQ-0160 Provide User Interface Services
QC System
•DMS-REQ-0097 Level 1 Data Quality Report Definition
•DMS-REQ-0099 Level 1 Performance Report Definition
•DMS-REQ-0314 Compute Platform Heterogeneity
•DMS-REQ-0318 Data Management Unscheduled Downtime
•DMS-REQ-0096 Generate Data Quality Report Within Specified
•DMS-REQ-0098 Generate DMS Performance Report Within
Time
•DMS-REQ-0100 Generate Calibration Report Within Specified
The contents of this document are subject to configuration control by the
Team.
56
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Science Algorithms
•DMS-REQ-0032 Image Differencing
•DMS-REQ-0033 Provide Source Detection Software
•DMS-REQ-0042 Provide Astrometric Model
•DMS-REQ-0043 Provide Calibrated Photometry
•DMS-REQ-0052 Enable a Range of Shape Measurement Approaches
•DMS-REQ-0296 Pre-cursor, and Real Data
•DMS-REQ-0308 Software Architecture to Enable Community
Science Payloads
Alert Production
•DMS-REQ-0333 Maximum Likelihood Values and Covariances
•DMS-REQ-0347 Measurements in catalogs
•DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
•DMS-REQ-0010 Difference Exposures
•DMS-REQ-0074 Difference Exposure Attributes
•DMS-REQ-0069 Processed Visit Images
•DMS-REQ-0029 Generate Photometric Zeropoint for Visit
•DMS-REQ-0030 Generate WCS for Visit Images
•DMS-REQ-0070 Generate PSF for Visit Images
•DMS-REQ-0072 Processed Visit Image Content
•DMS-REQ-0327 Background Model Calculation
•DMS-REQ-0328 Documenting Image Characterization
•DMS-REQ-0097 Level 1 Data Quality Report Definition
•DMS-REQ-0266 Exposure Catalog
•DMS-REQ-0269 DIASource Catalog
•DMS-REQ-0270 Faint DIASource Measurements
•DMS-REQ-0271 DIAObject Catalog
•DMS-REQ-0272 DIAObject Attributes
•DMS-REQ-0274 Alert Content
•DMS-REQ-0317 DIAForcedSource Catalog
•DMS-REQ-0319 Characterizing Variability
•DMS-REQ-0324 Matching DIASources to Objects
•DMS-REQ-0321 Level 1 Processing of Special Programs
The contents of this document are subject to configuration control by the
Team.
57
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Alert Production (continued)
•DMS-REQ-0344 Constraints on Level 1 Special Program
ation
•DMS-REQ-0312 Level 1 Data Product Access
•DMS-REQ-0002 Transient Alert Distribution
•DMS-REQ-0285 Level 1 Source Association
•DMS-REQ-0288 Use of External Orbit Catalogs
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0032 Image Differencing
•DMS-REQ-0033 Provide Source Detection Software
•DMS-REQ-0042 Provide Astrometric Model
•DMS-REQ-0043 Provide Calibrated Photometry
•DMS-REQ-0052 Enable a Range of Shape Measurement
•DMS-REQ-0308 Software Architecture to Enable Community
Annual Calibration
•DMS-REQ-0059 Bad Pixel Map
•DMS-REQ-0060 Bias Residual Image
•DMS-REQ-0061 Crosstalk Correction Matrix
•DMS-REQ-0062 Illumination Correction Frame
•DMS-REQ-0063 Monochromatic Flatfield Data Cube
•DMS-REQ-0130 Calibration Data Products
•DMS-REQ-0132 Calibration Image Provenance
•DMS-REQ-0282 Dark Current Correction Frame
•DMS-REQ-0283 Fringe Correction Frame
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0289 Calibration Production Processing
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable Community
The contents of this document are subject to configuration control by the
Team.
58
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Daily Calibration Update
•DMS-REQ-0059 Bad Pixel Map
•DMS-REQ-0060 Bias Residual Image
•DMS-REQ-0061 Crosstalk Correction Matrix
•DMS-REQ-0062 Illumination Correction Frame
•DMS-REQ-0063 Monochromatic Flatfield Data Cube
•DMS-REQ-0130 Calibration Data Products
•DMS-REQ-0132 Calibration Image Provenance
•DMS-REQ-0282 Dark Current Correction Frame
•DMS-REQ-0283 Fringe Correction Frame
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0101 Level 1 Calibration Report Definition
•DMS-REQ-0100 Generate Calibration Report Within
•DMS-REQ-0131 Calibration Images Available Within
•DMS-REQ-0289 Calibration Production Processing
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable Community
Data Release Production
•DMS-REQ-0326 Storing Approximations of Per-pixel
•DMS-REQ-0331 Computing Derived Quantities
•DMS-REQ-0333 Maximum Likelihood Values and Covariances
•DMS-REQ-0347 Measurements in catalogs
•DMS-REQ-0325 Regenerating L1 Data Products During
cessing
•DMS-REQ-0034 Associate Sources to Objects
•DMS-REQ-0047 Provide PSF for Coadded Images
•DMS-REQ-0103 Produce Images for EPO
•DMS-REQ-0106 Coadded Image Provenance
•DMS-REQ-0267 Source Catalog
•DMS-REQ-0268 Forced-Source Catalog
•DMS-REQ-0275 Object Catalog
•DMS-REQ-0046 Provide Photometric Redshifts of Galaxies
•DMS-REQ-0276 Object Characterization
•DMS-REQ-0277 Coadd Source Catalog
•DMS-REQ-0349 Detecting extended low surface brightness
•DMS-REQ-0278 Coadd Image Method Constraints
•DMS-REQ-0279 Deep Detection Coadds
•DMS-REQ-0280 Template Coadds
•DMS-REQ-0281 Multi-band Coadds
•DMS-REQ-0329 All-Sky Visualization of Data Releases
•DMS-REQ-0330 Best Seeing Coadds
•DMS-REQ-0335 PSF-Matched Coadds
•DMS-REQ-0337 Detecting faint variable objects
The contents of this document are subject to configuration control by the
Team.
59
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Data Release Production (continued)
•DMS-REQ-0320 Processing of Data From Special
•DMS-REQ-0350 Associating Objects across data
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0032 Image Differencing
•DMS-REQ-0033 Provide Source Detection Software
•DMS-REQ-0042 Provide Astrometric Model
•DMS-REQ-0043 Provide Calibrated Photometry
•DMS-REQ-0052 Enable a Range of Shape Measurement
•DMS-REQ-0308 Software Architecture to Enable
MOPS and Forced Photometry
•DMS-REQ-0004 Nightly Data Accessible Within
•DMS-REQ-0273 SSObject Catalog
•DMS-REQ-0317 DIAForcedSource Catalog
•DMS-REQ-0319 Characterizing Variability
•DMS-REQ-0321 Level 1 Processing of Special
•DMS-REQ-0344 Constraints on Level 1 Special
ation
•DMS-REQ-0341 Providing a Precovery Service
•DMS-REQ-0089 Solar System Objects Available
•DMS-REQ-0286 SSObject Precovery
•DMS-REQ-0287 DIASource Precovery
•DMS-REQ-0288 Use of External Orbit Catalogs
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable
The contents of this document are subject to configuration control by the
Team.
60
LARGE SYNOPTIC SURVEY TELESCOPE
Data Management System DesignLDM-148
Latest Revision 2017-07-04
Component
Requirements
Periodic Calibration
•DMS-REQ-0059 Bad Pixel Map
•DMS-REQ-0060 Bias Residual Image
•DMS-REQ-0061 Crosstalk Correction Matrix
•DMS-REQ-0062 Illumination Correction Frame
•DMS-REQ-0063 Monochromatic Flatfield Data Cube
•DMS-REQ-0130 Calibration Data Products
•DMS-REQ-0132 Calibration Image Provenance
•DMS-REQ-0282 Dark Current Correction Frame
•DMS-REQ-0283 Fringe Correction Frame
•DMS-REQ-0265 Guider Calibration Data Acquisition
•DMS-REQ-0289 Calibration Production Processing
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable Community
Raw Calibration
•DMS-REQ-0101 Level 1 Calibration Report Definition
•DMS-REQ-0131 Calibration Images Available Within Specified
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable Community
Template Generation
•DMS-REQ-0280 Template Coadds
•DMS-REQ-0009 Simulated Data
•DMS-REQ-0308 Software Architecture to Enable Community
Science Primitives
•DMS-REQ-0042 Provide Astrometric Model
•DMS-REQ-0043 Provide Calibrated Photometry
•DMS-REQ-0052 Enable a Range of Shape Measurement Approaches
•DMS-REQ-0351 Provide Beam Projector Coordinate Calculation
16 References
[1]
LSST DM
https://developer.lsst.io/
Guide, URL
[2]
[LDM-139]
, Becla, J., Lim,
Data
K.T.,
Management
2013,
Storage Sizing
,LDM-139,
and I/O Model Explanation
[3][LDM-141],Becla,J.,Lim,K.T.,2013,DataManagementStorageSizingandI/OModel,LDM-141,URLhttps:
//ls.st/LDM-141
Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
Team.
61
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
[4][LDM-135],Becla,J.,Wang,D.,Monkewitz,S.,etal.,2013,DatabaseDesign,LDM-135,URLhttps://ls.st/
LDM-135
[5][LSE-29],Claver,C.F.,TheLSSTSystemsEngineeringIntegratedProjectLSSTSystemRequirements,
LSE-29,URLhttps://ls.st/LSE-29
[6][LSE-30],Claver,C.F.,TheLSSTSystemsEngineeringIntegratedProjectLSSTSystemRequirements,
LSE-30,URLhttps://ls.st/LSE-30
[7][LSE-72],Dubious-Felsmann,G.,Schumacher,G.,Selvy,B.,2014,OCSCommandDictionaryforData
ment,LSE-72,URLhttps://ls.st/LSE-72
[8][LSE-81],Dubois-Felsmann,G.,2013,LSSTScienceandProjectSizingInputs,LSE-81,URLhttps://ls.st/LSE-81
[9][LSE-61],Dubois-Felsmann,G.,2016,LSSTDataManagementSubsystemRequirements,LSE-61,URLhttps:
//ls.st/LSE-61
[10][LSE-82],Dubois-Felsmann,G.,Lim,K.T.,2013,ScienceandProjectSizingInputsExplanation,LSE-82,URL
https://ls.st/LSE-82
[11][LDM-542],Dubois-Felsmann,G.,Lim,K.T.,Wu,X.,etal.,2017,LSSTSciencePlatformDesign,LDM-542,URL
[12][LDM-129],Freemon,M.,Kantor,J.,2013,DataManagementInfrastructureDesign,LDM-129,URLhttps:
//ls.st/LDM-129
[13][LDM-143],Freemon,M.,Pietrowicz,S.,2013,SiteSpecificInfrastructureEstimation,LDM-143,
URLhttps://ls.st/LDM-143
[14][LDM-144],Freemon,M.,Pietrowicz,S.,Alt,J.,2016,SiteSpecificInfrastructureEstimation,LDM-144,
URLhttps://ls.st/LDM-144
[15][LPM-17],Ivezić,Ž.,TheLSSTScienceCollaboration,2011,LSSTScienceRequirementsDocument,LPM-17,URL
[16][LSE-163],Jurić,M.,etal.,2017,LSSTDataProductsDefinitionDocument,LSE-163,URLhttps://ls.st/LSE-163
[17][LDM-142],Kantor,J.,2017,NetworkSizingModel,LDM-142,URLhttps://ls.st/LDM-142
[18][LDM-138],Kantor,J.,Axelrod,T.,Lim,K.T.,2013,DataManagementComputeSizingModel,LDM-138,URL
[19][LSE-78],Lambert,R.,Kantor,J.,Huffer,M.,etal.,2017,LSSTObservatoryNetworkDesign,LSE-78,URLhttps:
[20][LDM-230],Lim,K.T.,2013,AutomatedOperationoftheLSSTDataManagementSystem,LDM-230,URLhttps:
//ls.st/LDM-230
[21][LDM-140],Lim,K.T.,Smith,C.,Axelrod,T.,Dubois-Felsmann,G.,Freemon,DataManagement
ComputeSizingExplanation,LDM-140,URLhttps://ls.st/LDM-140
Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
Team.
62
LARGESYNOPTICSURVEYTELESCOPE
DataManagementSystemDesignLDM-148LatestRevision2017-07-04
[22][LDM-148],Lim,K.T.,Bosch,J.,Dubois-Felsmann,G.,etal.,2017,DataManagementSystemDesign,LDM-148,
URLhttps://ls.st/LDM-148
[23][LDM-152],Lim,K.T.,Dubois-Felsmann,G.,Johnson,M.,Jurić,M.,Petravick,DataManagement
MiddlewareDesign,LDM-152,URLhttps://ls.st/LDM-152
[24][Document-5373],Pinto,P.,Kantor,J.,Strauss,M.,Sweeney,D.,2008,DataAccessWhitePaper,Document-
5373,URLhttps://ls.st/Document-5373
[25][LDM-151],Swinbank,J.D.,etal.,2017,DataManagementSciencePipelinesDesign,LDM-151,URLhttps:
//ls.st/LDM-151
Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
Team.
63
Back to top