LargeSynopticSurveyTelescop
    DataManagementSystem
    K.-T.Lim,J.Bosch,G.Dubois-Felsmann,
    W.O’Mullane,D.Petravick,andtheDMLeadership
    LDM-148
    LatestRevision:2017-07-04
    ThisLSSTdocumenthasbeenapprovedasaContent-ControlledDocument
    nicalControlTeam.Ifthisdocumentischangedorsuperseded,
    theHandledesignationshownabove.Thecontrolisonthemost
    thisHandleintheLSSTdigitalarchiveandnotprintedversions.
    foundinthecorrespondingDMRFC.
    LARGESYNOPTICSURVEYTELESCOPE

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    Abstract
    TheLSSTDataManagementSystem(DMS)isasetofservices
    ofsoftwarecomponentsrunningoncomputationalandnetworking
    thatcombinetodeliversciencedataproductstotheobservatory’s
    portobservatoryoperations.Thisdocumentdescribes
    viceinstances,andtheirdeploymentenvironmentsas
    them,therestoftheLSSTsystem,andtheoutsideworld.
    Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
    Team.
    ii

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    ChangeRecord
    VersionDateDescriptionOwnername
    22011-08-09CopiedfromMREFCProposalintoLDM-148
    handle,reformatted
    RobertMcKercher
    32011-08-15UpdatedforPreliminaryDesignReviewTimAxelrod,K-TLim,
    MikeFreemon,Jeffrey
    Kantor
    42013-10-09UpdatedforFinalDesignReviewMarioJuric,K-TLim,Jef-
    freyKantor
    5.02017-07-04RewrittenforConstructionandOperations.
    ApprovedinRFC-358.
    K-TLim
    Documentcurator:Kian-TatLim
    Documentsourcelocation:
    https://github.com/lsst/LDM-148
    The contents of this document are subject to configuration control by the
    Team.
    iii

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Contents
    1 Introduction
    1
    2 Summary Concept of Operations
    2
    3 Sizing
    4
    4 Component Overview
    8
    5 Base Enclave
    14
    5.1 Service Descriptions
    ........................
    16
    5.1.1 Image and EFD
    .....................
    Archiving
    16
    5.1.2 Prompt Processing
    .....................
    Ingest
    16
    5.1.3 OCS Driven Batch
    .....................
    Control
    17
    5.1.4 Telemetry
    .......................
    Gateway
    17
    5.2 Interfaces
    ............................
    17
    6 NCSA Enclave Level 1 Domain
    18
    6.1 Service Descriptions
    ........................
    19
    6.1.1 Prompt Processing
    .......................
    19
    6.1.2 OCS Driven Batch
    ....................
    Processing
    19
    6.1.3 Offline Processing
    .......................
    20
    6.1.4 Level 1 Quality
    ......................
    Control
    20
    6.1.5 Alert Distribution
    .......................
    20
    The contents of this document are subject to configuration control by the
    Team.
    iv

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    6.1.6 Alert........................
    Filtering
    20
    6.2 Interfaces
    ............................
    21
    7 NCSA Enclave Level 2 Domain
    21
    7.1 Service Descriptions
    ........................
    22
    7.1.1 Template and Calibration Products
    ......
    Production
    22...
    7.1.2 Data Release Production
    ..................
    Execution 22
    7.1.3 Level 2 Quality
    ......................
    Control
    23
    7.2 Interfaces
    ............................
    23
    8 Satellite Computing Enclave
    23
    8.1 Service Description
    .........................
    23
    8.1.1 Data Release Production
    .........
    Satellite Processing
    23
    .....
    8.2 Interfaces
    ............................
    23
    9 Data Access Center Enclaves
    24
    9.1 Service Descriptions
    ........................
    24
    9.1.1 Bulk Data......................
    Distribution
    24
    9.1.2 LSST Science Platform
    .................
    DAC instances24
    9.2 Interfaces
    ............................
    25
    10 NCSA Enclave Analysis and Developer Support26
    Domain
    10.1 Service Descriptions
    ........................
    26
    10.1.1 LSST Science Platform Science
    ........
    Validation
    26
    .....
    instance
    The contents of this document are subject to configuration control by the
    Team.
    v

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    10.1.2 Developer
    .......................
    Services
    26
    10.1.3 Integration
    .....................
    and Testing
    26
    10.2 Interfaces
    ............................
    27
    11 Commissioning Cluster
    27
    11.1 Service Description
    .........................
    27
    11.1.1 LSST Science Platform.........
    Commissioning
    27
    .....
    instance
    11.2 Interfaces
    ............................
    28
    12 Infrastructure Domain
    28
    12.1 Service Descriptions
    ........................
    28
    12.1.1 Data Backbone
    ........................
    28
    12.1.2 Management/Monitoring
    .....................
    30
    12.1.3 Provisioning/Deployment
    .....................
    30
    12.1.4 Workload/Workflow
    ......................
    31
    12.1.5 Batch Processing
    .......................
    31
    12.1.6 Identity......................
    Management
    31
    12.2 Interfaces
    ............................
    31
    13 Software Components
    32
    13.1 Science.........................
    Payloads
    32
    13.1.1 Alert Production
    .....................
    Payload
    32
    13.1.2 MOPS.Payload
    .......................
    32
    The contents of this document are subject to configuration control by the
    Team.
    vi

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    13.1.3 Raw Calibration..................
    Validation Payload32
    13.1.4 Daily Calibration Products
    ..........
    Update Payload
    33
    ......
    13.1.5 Periodic Calibration Products
    ........33
    Production
    .....
    Payload
    13.1.6 Template Generation
    ...................
    Payload
    33
    13.1.7 Annual Calibration Products
    ........33
    Production
    .....
    Payload
    13.1.8 Data Release Production
    ..................
    Payload
    33
    13.2 SUIT
    .............................
    34
    13.3 Middleware
    ...........................
    34
    13.3.1 Data Butler
    .....................
    Access Client
    34
    13.3.2 Parallel Distributed
    .................
    Database (Qserv)
    34
    13.3.3 Task Framework
    .......................
    34
    14 Design and Implementation Standards
    35
    14.1 Python.Version
    .........................
    35
    14.2 HTTPS Protocol
    ..........................
    35
    15 Appendix: Traceability
    37
    15.1 Requirement to Component
    ..................
    Traceability 37
    15.2 Component to Requirement
    ..................
    Traceability 45
    16 References
    61
    The contents of this document are subject to configuration control by the
    Team.
    vii

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Data Management System Design
    1 Introduction
    The purpose of the LSST Data Management System (DMS) is to
    to the observatory’s users and to support observatory operations.
    vices employing a variety of software components running
    infrastructure. The DMS is constructed by the DM subsystem
    Operations era, it is operated by a combination of the LSST
    and Observatory Operations departments.
    The data products to be delivered are defined and described
    Document
    LSE-163
    (). These are divided into three major categories.
    One category of data products is generated on a nightly or
    processed/calibrated, and difference images as well as alerts
    able objects detected from the images, published within
    able catalogs. These data products can be considered “online”,
    by the observing cadence of the observatory. This category
    as “Level 1”. These products are intended to enable detection
    time-domain events.
    A second category of data products is generated on an ann
    complete reprocessing of the set of images taken to date
    containing measurements and characterization of tens of
    high and uniform astrometric and photometric accuracy.
    the first category of data products is regenerated, often
    category also includes other data products such as calibration
    are generated in an “offline” mode, not directly tied to the
    has historically been referred to as “Level 2”, including
    the first category.
    The third category of data products is not generated by the
    created, or imported by science users for their own science
    value from their close association with or derivation from
    The contents of this document are subject to configuration control by the
    Team.
    1

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    is responsible for providing facilities, services, and
    This category has historically been referred to as “Level
    Data products are delivered to science users through Data
    streams of near-realtime alerts and telescope pointing
    data product has associated metadata providing provenance
    ing it to relevant calibration information in the archive.
    but significant computational, storage, networking, and
    a flexible, multi-tenant environment for professional astronomers
    trieve, manipulate, and annotate LSST data products in order
    and inquiry.
    The first section of this document describes how the DMS components
    erate and distribute the data products. The next section
    computing environments was estimated. Subsequent sections
    ponents of the DMS in more detail, including their interfaces
    subsystems, and with the outside world.
    2 Summary Concept of Operations
    The principal functions of the DMS are to:
    •Process the incoming stream of images generated by the
    ing by archiving raw images, generating transient alerts,
    and object catalogs.
    •Periodically (at least annually) process the accumulated
    photometric and astrometric calibration, measure the
    characterize the time-dependent behavior of objects.
    run form a data release (DR), which is a static, self-consistent
    ing scientific analysis of LSST data and publication of
    archived for the entire operational life of the LSST archive.
    •Periodically create new calibration data products, such
    be used by the other processing functions.
    The contents of this document are subject to configuration control by the
    Team.
    2

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    •Make all LSST data available through an interface that
    extent, community-based standards such as those being
    servatory (VO) in collaboration with the International
    Provide enough processing, storage, and network bandwidth
    the data without petabyte-scale data transfers.
    The latency requirements for alerts determine several aspects
    cost. An alert is triggered by an unexpected excursion in
    appearance of a previously undetected object such as a supernova
    ical time scale of some of these events may warrant follow-up
    time scales. These excursions in brightness must be recognized
    sulting alert data product sent on its way, within 60 seconds.
    the decision to acquire high-bandwidth/high-reliability
    mit at Cerro Pachon to the Base in La Serena and from Chile
    the significant computational resources necessary for promptly
    to be located in cost-effective locations: the Base has far
    and rack space capacity than the Summit, and placing the scientific
    for far greater flexibility in the allocation of resources
    forming cross-talk correction on the data in the data acquisition
    alert processing at the amplifier and CCD levels, where possible,
    latency to alert delivery.
    The Data Release processing requires extensive computation,
    images of an object in order to measure it as accurately as
    and workflow management system and Task Framework are used
    into manageable units of work that can be assigned to available
    dedicated processing clusters at NCSA and CC-IN2P3.
    Calibration data products must be created and updated at
    Data Release periods. The stability of the system is expected
    annual calibration productions. The daily production must
    tory schedule, occurring after raw calibration frames have
    observing is planned. This requirement necessitates the
    Observatory Control System to trigger remote calibration
    The DACs are a key component of the DMS, giving the community
    The contents of this document are subject to configuration control by the
    Team.
    3

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    face to interact with and utilize the LSST data products
    the LSST Science Platform (LSP) is deployed in each DAC to
    tal, JupyterLab (notebook), and Web API aspects. Substantial
    bandwidth is devoted to ensuring that the LSP is responsive
    vast LSST data products.
    Underlying all of the above is a Data Backbone that provides
    for all LSST data products. The Data Backbone links all of
    Data Access Centers, acting as the spine that supports them
    3 Sizing
    A fundamental question is how large the LSST Data Management
    end, a complex analytical model has been developed driven
    specifications. Specifications from the science requirements
    and the observing strategy, translate directly into numbers
    nomical objects, and ultimately into required network
    systems. Specific science requirements of the survey determine
    be maintained in the DMS products, which in turn determine
    and the computer power necessary to execute them. The relationship
    model and their flow-down from systems and DMS1.
    requirements
    De-
    tailed sizing computations and associated explanations
    the Figure.
    Key input parameters include camera characteristics, the
    the number of observed stars and galaxies expected per band,
    data element, the data transfer rates between and within
    query rates of input and output data, the alert generation
    requirements for all data products.
    Processing requirements were extrapolated from the functional
    type pipelines and algorithms, and existing precursor pipelines
    part of every Data Release, all data previously processed
    gorithms, calibration products, and parameters. This causes
    increase with time. Advances in hardware performance, however,
    The contents of this document are subject to configuration control by the
    Team.
    4

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    DM Compute Sizing
    LDM-138
    Explanation:
    LDM-140
    DM Storage & I/O Sizing
    LDM-141
    Explanation:
    LDM-139
    Network Sizing
    LDM-142
    Explanation:
    LSE-78
    DM Site Specific Infrastructure
    Architecture/Design/Costs
    LDM-144
    Explanation:
    LDM-143
    LSST Science Requirements
    LPM-17
    LSST System Requirements
    LSE-29
    Observatory System Specs
    LSE-30
    DM Subsystem Requirements
    LSE-61
    Data Products Definition Document
    LSE-163
    Community Access White Paper
    Document-5373
    LSST Science and
    Project Sizing Inputs
    LSE-81
    Explanation:
    LSE-82
    DM System Design
    LDM-148
    DM Applications Design
    LDM-151
    DM Middleware Design
    LDM-152
    DM Database Design
    LDM-135
    DM Infrastructure Design
    LDM-129
    Figure1:DMSInfrastructureSizingandEstimation.
    ThecontentsofthisdocumentaresubjecttoconfigurationcontrolbytheLSSTDMTechnicalControl
    Team.
    5

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    Table1:DMSComputeInfrastructureSizing;growthfromSurvey
    high-watermark
    ArchiveSiteBaseSite
    Compute
    TeraFLOPS(sustained)197→97030→62
    Nodes436→305(455hwm)56→17(59hwm)
    Cores18K→62K3K→4K
    Database
    TeraFLOPS(sustained)40→31055→306
    Nodes94→113(148hwm)108→98(133hwm)
    Facilities
    FloorSpace826→744ft
    2(834hwm)278→195ft2(435hwm)
    Power274→273kW(309hwm)158→248kW(248hwm)
    Cooling0.9→0.9mmbtu(1.1hwm)0.5→0.8mmbtu(0.8hwm)
    Table2:DMSStorageInfrastructureSizing;growthfromSurvey
    ArchiveSiteBaseSite
    FileStorage
    Capacity24→81PB
    Drives1602→862597→249
    Bandwidth493→714GB/s(752hwm)223→231GB/s(236hwm)
    Database
    Capacity29→99PB16→72PB
    Drives3921→22882190→1642
    Bandwidth1484→2040GB/s(2163hwm)829→1169GB/s(1615hwm)
    TapeStorage
    Capacity31→242PB
    Tapes2413→3691(4117hwm)
    TapeBandwidth36→65GB/s
    numberofnodesneededandthepowerandcoolingdevoted
    theperformancefiguresinTable1toreachahigh-watermarkduringthesurvey.
    Storageandinput/outputrequirementswereextrapolated
    products,theDMSandprecursordatabaseschemas,andexisting
    systemoverheadfactorsinprecursorsurveysandexperiments
    summaryofkeynumbersisinTable2.
    Communicationsrequirementsweredevelopedandmodeled
    query/responseload,extrapolatedfromexistingsurveys
    requirementsareillustratedinFigure2forper-visittransfers.Peakbandwidths
    secondbudgetforSummittoBaseimagetransferanda5second
    imagetransfer.
    TheSummittoBaseandBasetoNCSAnetworklinkshavebeen
    Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
    Team.
    6

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    forfourmainreasons:first,becausetheincrementalcosts
    hasbeenprovisionedatallhavebeensmall;second,toallow
    withtheCameraDataSystemorperformingimageanalysis
    alerts,tobeperformedinappropriatelocations;third,
    fourth,tosimplifycertaincomponentssuchastheForwarders
    withordependonthenetworks.Ahigh-speedBasetoNCSA
    tobetransferredsouthtotheChileanDACoverthenetwork
    media,asoriginallyplanned,decouplingsciencefrommaintenance
    agreaterextent.
    Observatory
    Prompt
    Processing
    Ingest
    Image and
    EFD
    Archiving
    Data
    Backbone
    Endpoint
    Prompt
    Processing
    Alert
    Distribution
    Alert
    Filtering
    Camera Data
    Raw
    System (DAQ)
    Images
    24 GB
    Crosstalk-
    Corrected
    Images
    24 GB
    Data
    Backbone
    Endpoint
    Processed Visit Image
    & Difference Image
    36 GB each
    Catalog Entries
    72 MB
    Alert Stream
    1.3 GB
    Full-Feed
    Alert
    Broker
    Alert Stream
    1.3 GB
    Alert Client
    Filtered
    Alert Stream
    1.3 MB
    Image size estimates are per visit of two snap exposures,
    uncompressed, either 18 bits per pixel transferred in 32 bits
    or 3×32 bits per pixel (image, mask, variance). Alternate
    Science Visits of one 30-second exposure would produce
    one 12 GB raw image and one 12 GB crosstalk-corrected
    image.
    Alert stream and catalog estimates are for max
    40K alerts per visit. Alerts include postage stamp images
    and one year DIASource history, uncompressed.
    Filtering ratio is for illustration only.
    Base
    NCSA
    NCSA
    External
    Science
    Users
    LSST
    Science
    Platform
    DAC
    NCSA
    Figure
    Near
    2:
    Real-Time Data Flows
    In all of the above, industry-provided
    LDM-143
    ) were
    technology
    used to extrapolate
    trends
    to the LSST construction and operations phases in which the
    figured, deployed, operated, and maintained. A just-in-time
    to leverage favorable cost/performance trends.
    The contents of this document are subject to configuration control by the
    Team.
    7

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    The resulting performance and sizing requirements show
    class system with correspondingly large data input/output
    spite this size, technology trends show this to be well within
    commodity-based systems during the construction and operations
    4 Component Overview
    The services that make up the DMS are in turn made up of software
    components, instantiated in a particular configuration in
    to perform a particular function. Some software components
    ers are general-purpose and reused across multiple services.
    instance in the production system; others have several,
    the development and integration environments for testing
    The DMS services can be considered to consist of four tiers
    tier is the LSST Science Platform, which is deployed in the
    vironments to provide a user interface and analysis environment
    staff. The detailed design of this tier
    LDM-542
    is given
    . Thein
    next
    “LSST
    tier
    Science
    is composed of science “applications” software that generates
    is used to build “payloads”, sequences of pipelines, that
    product generation tasks. It is also used by science users
    ucts. The detailed design of the components in this tier is
    PipelinesLDM-151
    Design,
    . A lower tier is “middleware” software components
    that execute the science application payloads and isolate
    cluding changes to underlying technologies. These components
    science users and staff. The detailed design of the components
    Management Middleware
    LDM-152
    .Design,
    The bottom tier is “infrastructure”:
    networking, and low-level software and services that provide
    detailed design of components in this tier is given in Data
    ture,
    LDM-129
    , and Network
    LSE-78
    Design,
    .
    The DMS computing environments reside in four main physical
    including the main Observatory and Auxiliary Telescope
    the Base Facility data center located at the Base Site in
    data center at the National Center for Supercomputing Applications
    The contents of this document are subject to configuration control by the
    Team.
    8

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    USA; and the Satellite Computing Facility at CC-IN2P3 in
    high-speed networks to allow rapid data movement. The Base
    production computational environments (the Base Enclave
    and also the US and Chilean Data Access Centers. In addition,
    putational environment also resides at the Base Facility.
    The DMS service instances in the NCSA Enclave can be broken
    domains: a near-realtime online domain (L1) closely linked
    offline Level 2 domain (L2) organized primarily around the
    and an analysis and developer support domain (ADS) encompassing
    ations staff use for science validation, software development,
    testing. In addition, an underlying infrastructure domain
    of the other domains, including a common Data Backbone that
    archiving and that is the primary connection between all
    distinguished by having different users, operations timescales,
    nents.
    The service instances that make up the DMS include (with the
    domains they are in noted):
    •Image and EFD Archiving (Base)
    •Prompt Processing Ingest (Base)
    •Observatory Control System (OCS) Driven Batch Control
    •Telemetry Gateway (Base)
    •Prompt Processing (L1)
    •OCS Driven Batch Processing (L1)
    •Offline Processing (L1)
    •Alert Distribution (L1)
    •Alert Filtering (L1)
    •Level 1 Quality Control (QC) (L1)
    •Template and Calibration Products Production Execution
    The contents of this document are subject to configuration control by the
    Team.
    9

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    •Data Release Production Execution (L2)
    •Data Release Production Satellite Processing (Satellite
    •Level 2 QC (L2)
    •LSST Science Platform Commissioning Cluster instance
    •LSST Science Platform Data Access Center instances (DACs)
    •Bulk Data Distribution (DAC)
    •LSST Science Platform Science Validation instance (ADS)
    •Developer Services (ADS)
    •Integration and Test (ADS)
    •Data Backbone (Infra)
    •Management/Monitoring(Infra)
    •Provisioning/Deployment(Infra)
    •Workload/Workflow(Infra)
    •HTCondor Batch Processing (Infra)
    •Identity Management (Infra)
    The relationships between these services, their deployment
    mains, and science application “payloads”
    3.
    can be visualized
    The common infrastructure services
    4.
    are illustrated in Figure
    The science application software for the Alert Production,
    Production, and calibration processing is built out of a
    In turn, those frameworks build on middleware that provides
    relationships between the packages implementing these se
    underlying middleware packages
    5.
    are shown in Figure
    Key applications software components include:
    The contents of this document are subject to configuration control by the
    Team.
    10

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Data Backbone
    NCSA Enclave
    Analysis and
    Developer Support
    Level 1
    Level 2
    Base Enclave
    Prompt
    Processing
    Ingest
    Offline
    Processing
    OCS
    Driven
    Batch Ctrl
    Image and
    EFD
    Archiving
    Level 1
    Quality
    Control
    Alert
    Distribution
    Telemetry
    Gateway
    Alert
    Filtering
    Template &
    Calib. Prod.
    Production
    Data
    Release
    Production
    Level 2
    Quality
    Control
    US Data Access Center
    Bulk Data
    Distribution
    Science
    Platform
    (DAC)
    Observatory
    Developer
    Services
    Integration
    & Test
    Science
    Platform
    (Sci. Valid.)
    Data
    Backbone
    Endpoint
    Prompt
    Processing
    Data
    Backbone
    Endpoint
    OCS Batch
    Processing
    Data
    Backbone
    Endpoint
    Commissioning Cluster
    Science
    Platform
    (Commiss.)
    Data
    Backbone
    Endpoint
    RabbitMQ
    BBFTP
    HTCondor
    Satellite Processing CC-IN2P3
    DRP
    Satellite
    Processing
    Pegasus /
    HTCondor
    Tape
    Periodic
    Calibration
    Payload
    Template
    Generation
    Payload
    Raw Calib
    Validation
    Payload
    Alert
    Production
    Payload
    Annual
    Calibration
    Payload
    DRP
    Payload
    MOPS
    Payload
    Daily Cal.
    Update
    Payload
    Chilean Data Access Center
    Science
    Platform
    (DAC)
    Data
    Backbone
    Endpoint
    Science
    Users
    Staff
    Staff
    Alert
    Users
    Community
    Alert
    Brokers
    EPO
    Other Data
    Partners
    Data
    Backbone
    Endpoint
    Figure
    Data
    3:
    Management System Deployment
    The contents of this document are subject to configuration control by the
    Team.
    11

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    NCSA Enclave
    Base Enclave
    US Data Access Center
    Commissioning Cluster
    Satellite Processing CC-IN2P3
    Chilean Data Access Center
    Workload /
    Workflow
    Provisioning /
    Deployment
    HTCondor
    vSphere +
    Puppet
    Management /
    Monitoring
    Workload /
    Workflow
    Provisioning /
    Deployment
    HTCondor
    Management /
    Monitoring
    Workload /
    Workflow
    Provisioning /
    Deployment
    HTCondor
    Management /
    Monitoring
    Management /
    Monitoring
    Workload /
    Workflow
    Provisioning /
    Deployment
    HTCondor
    Management /
    Monitoring
    Workload /
    Workflow
    Provisioning /
    Deployment
    HTCondor
    Management /
    Monitoring
    Identity
    Management
    Identity
    Management
    Replica
    Figure
    Data
    4:
    Management System Common Infrastructure Services
    The contents of this document are subject to configuration control by the
    Team.
    12

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    •Low-level astronomical software primitives
    afw
    )
    and data
    •Image processing and measurement framework
    ip_*
    ,
    meas_*
    )with core
    •Additional image processing and
    meas_extensions_*
    measurement
    )
    algorithms
    •High-level algorithms and driver
    pipe_tasks
    scripts
    ,
    pipe_drivers
    that
    ) define pipelines
    •Camera-specific customizations
    obs_*
    )
    (
    Figure
    Data
    5:
    Management Science Pipelines Software “Stack”
    Key middleware components include:
    The contents of this document are subject to configuration control by the
    Team.
    13

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    •Data access client
    daf_persistence
    (Data Butler) (
    •Parallel distributed
    qserv
    )
    database (
    •Task framework
    pex_*
    ,
    log
    ,
    pipe_base
    (
    ,
    ctrl_pool
    )
    •Workflow and orchestration for
    ctrl_*
    production
    )
    control (
    Infrastructure components include:
    •Other databases (typically relational)
    •Filesystems
    •Authentication and authorization (identity management)
    •Provisioning and resource management
    •Monitoring
    The relationships between the middleware and infrastructure
    Figure
    6.
    5 Base Enclave
    Services located in this enclave are located at the Base solely
    the OCS or the Camera Data System (also known as the Camera
    cases, services located here interact closely with corresponding
    Level 1 Domain, to the point where the Base service cannot
    operational. This reliance has been taken into account in
    The primary goals of the services in this enclave are to transfer
    either to NCSA, from NCSA, or to the Data Backbone.
    The services in this enclave and their partners in the NCSA
    to run rapidly and reliably. They run at times (outside office
    The contents of this document are subject to configuration control by the
    Team.
    14

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Task Framework
    Config API
    Logging API
    MultiNodeTask
    API
    MultiCoreTask
    API
    Control and
    Orchestration
    Authentication
    & Authorization
    Provisioning &
    Resource
    Management
    Monitoring
    Data Butler
    Science Algorithms
    SuperTask
    Task
    Command Line
    Activator
    SUIT Activator
    Pre-Flight and
    Job-Level
    Activators
    Data Backbone
    RDBMS
    Qserv
    Files
    Figure
    Data
    6:
    Management Middleware and Infrastructure
    The contents of this document are subject to configuration control by the
    Team.
    15

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    are not amenable to a human-in-the-loop design. Instead,
    tonomously, often under the control of the OCS, with human
    trol only at the highest level.
    5.1 Service Descriptions
    Detailed concepts of operations for each service can be found
    the LSST Production
    LDM-230
    Services”
    ).
    (
    5.1.1 Image and EFD Archiving
    This component is composed of several Image Archiving service
    instances: one pair each for the LSSTCam, the ComCam, and
    graph, all of which may be operated simultaneously. These
    camera, including the wavefront sensors and the guide sensors
    when so configured, retrieving them from their respective
    They also capture specific sets of metadata associated with
    values and event timings, from the OCS publish/subscribe
    The image pixels and metadata are then permanently archived
    catch-up versions archive into the Data Backbone any raw
    missed by the primary archiving services due to network
    from the flash storage in the Camera Data System instances
    This component also includes an EFD Transformation service
    (including telemetry, events, configurations, and commands)
    annex, transforms it into a form more suitable for querying
    into the permanently archived “Transformed EFD” database
    5.1.2 Prompt Processing Ingest
    This component is composed of two instances that capture
    the LSSTCam and ComCam Camera Data Systems along with selected
    and/or EFD and transfer them to the Prompt Processing service
    Domain.
    The contents of this document are subject to configuration control by the
    Team.
    16

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    There is no Prompt Processing Ingest instance for the auxiliary
    5.1.3 OCS Driven Batch Control
    This service receives commands from the OCS and invokes the
    service in the NCSA Enclave Level 1 Domain to execute corresponding
    used for modest-latency analysis of images during Commissioning
    calibration images in normal observing operations. A summary
    formed is returned to the OCS for each command, following
    protocol.
    5.1.4 Telemetry Gateway
    This service obtains information from the NCSA Enclave Level
    and quality metrics from Prompt Processing of images and
    and transmits it to the OCS as specified in the Data Management-OCS
    tion Interface
    LSE-72
    ). Note
    (
    that more detailed information on the
    of DMS services will also be available to Observatory operators
    inated from the Management/Monitoring infrastructure
    environments.
    5.2 Interfaces
    OCS to all Base Enclave services: these interface through
    subsystem.
    Archiver and Catch-Up Archiver to Data Backbone: files are
    age via a file transfer mechanism, and their information
    Data Backbone management dataabases. The Data Butler is
    science-payload interface.
    EFD to EFD Transformer: this interface is via connection
    EFD as well as file transfer from the EFD’s Large File Annex.
    EFD Transformer to Data Backbone: Transformed EFD entries
    The contents of this document are subject to configuration control by the
    Team.
    17

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    formed EFD” database resident within the Data Backbone.
    Camera Data System to Archiver, Catch-Up Archiver, Prompt
    face through the custom library provided by the Camera Data
    Prompt Processing Ingest to Prompt Processing: BBFTP is
    ternational network from the ingest service to the processing
    OCS Driven Batch Control to OCS Driven Batch Processing:
    cution instructions over the international network from
    service and return status and result information.
    Telemetry Gateway from NCSA Enclave Level 1 Domain services:
    status and quality metrics to the gateway over the international
    6 NCSA Enclave Level 1 Domain
    This domain is responsible for the compute-intensive processing
    tions and other operations closely tied with the Observatory.
    images and metadata from the Observatory into “online” science
    them to the DACs, alert subscribers, and back to the OCS.
    The Prompt Processing, OCS Driven Batch Processing, and
    port execution of science payloads in three different ways.
    tegrated with the observing cadence and is intended to function
    result deadlines. OCS Driven Batch Processing is invoked
    latency requirements. Offline Processing is not invoked by
    control, typically during the daytime.
    The Alert Distribution and Alert Filtering services receive
    Processing of each scienc visit; they then provide alert
    and LSST data rights holders, respectively.
    The Level 1 Quality Control service monitors the “online”
    alerts, notifying operators if any anomalies are found.
    The contents of this document are subject to configuration control by the
    Team.
    18

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Like the services in the Base Enclave, these services need
    are designed to execute autonomously.
    6.1 Service Descriptions
    Detailed concepts of operations for each service can be found
    the LSST Production
    LDM-230
    Services”
    ).
    (
    6.1.1 Prompt Processing
    This service receives crosstalk-corrected images and metadata
    Ingest service at the Base and executes the Alert Production
    erating “online” data products that are stored in the Data
    payload then sends alerts to the Alert Distribution service.
    The Prompt Processing service has calibration (including
    science, and deep drilling modes. In calibration mode,
    tion payload that provides rapid feedback of raw calibration
    mode, two consecutive exposures are grouped and processed
    of exposure groupings to be processed as visits in deep drilling
    The service is required to deliver Alerts within 60 seconds
    standard science visit with 98% reliability.
    There is no Prompt Processing service instance for the Auxiliary
    6.1.2 OCS Driven Batch Processing
    This service executes science payloads in response to commands
    Control service at the Base and thus indirectly from the
    used for modest-latency analysis of images during Commissioning
    calibration images in normal observing operations. Images
    the Data Backbone, and results are provided back to the Data
    connection from this service to the Camera Data System. This
    latency from image acquisition to processing start by the
    Data Backbone transfer. A summary status for the processing
    The contents of this document are subject to configuration control by the
    Team.
    19

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Driven Batch Control service to be returned to the OCS.
    6.1.3 Offline Processing
    This service executes science payloads to ensure that all
    within 24 hours. In particular, this service executes the
    System payload. It also may execute a variant of the Alert
    Processing service encounters difficulties. Images and
    Backbone, and results are provided back to the Data Backbone.
    6.1.4 Level 1 Quality Control
    This service collects information on Level 1 science and
    processes the science data products from the Data Backbone
    surements, and monitors the measurement values against
    automated quality control capability for potentially detecting
    telescope, camera, data acquisition, or data processing.
    crossings are delivered to Observatory operators and to
    tists for verification, analysis, and resolution.
    6.1.5 Alert Distribution
    This service obtains alerts generated by the Alert Production
    them to community alert brokers and to the Alert Filtering
    6.1.6 Alert Filtering
    This service obtains an alert feed from the Alert Broker
    LSST data rights holders to execute limited filters against
    then distributed to the individuals.
    The contents of this document are subject to configuration control by the
    Team.
    20

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    6.2 Interfaces
    Prompt Processing to Alert Distribution and Alert Filtering:
    transport system.
    Prompt Processing to Offline Processing: in the event that
    lotted time window, processing can be cancelled and the failure
    Processing will redo the processing at a later time. Note
    computational resources have been provisioned, for the
    continue to run, with spare capacity used to maintain latency
    there would effectively be an infinite time window.
    Science Payloads to Data Backbone: payloads use the Data
    and catalog databases within the Data Backbone.
    7 NCSA Enclave Level 2 Domain
    This domain is responsible for all longer-period data processing
    largest and most complex payloads supported by the DMS: the
    tion (DRP) and periodic Calibration Products Productions
    even while the annual DRP is executing, hence the need for
    Quality Control Service monitors the science data products,
    lies are found.
    The services in this domain need to run efficiently and reliably
    ning weeks or months. They need to execute millions or billions
    becomes available while tracking the status of each and
    signed to execute autonomously with human oversight, monitoring,
    the highest level, although provisions are made for manual
    sary.
    This domain does not have direct users (besides the operators
    within it obtain inputs from the Data Backbone and place
    bone.
    The contents of this document are subject to configuration control by the
    Team.
    21

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    7.1 Service Descriptions
    7.1.1 Template and Calibration Products Production Execution
    This service executes various CPP science payloads at intervals
    tion Images and populate the Calibration Database with
    of raw calibration images from the Data Backbone and information
    This includes the computation of crosstalk correction matrices.
    product, the templates used by Alert Production are also
    raw science images from the Data Backbone. Additional information
    logs are also taken from the Data Backbone. The intervals
    depend on the stability of Observatory systems but are expected
    and annual executions. The annual execution is a prerequisite
    of the Data Release Production. The service involves human
    termine initial configurations of the payload, to monitor
    to provide additional configuration information during execution.
    7.1.2 Data Release Production Execution
    This service executes the DRP science payload annually to
    after the annual CPP is executed. A small-scale (about
    executed first to ensure readiness, followed by the full
    taken from the Data Backbone along with Master Calibration
    the Transformed EFD. Additional information such as external
    from the Data Backbone. Computing is performed in conjunction
    cessing service at CC-IN2P3, which will have capacity for
    data products from both the mini-production and the main
    Data Backbone, including both images and catalogs. From
    staff scientists and selected external scientists using
    LSST Science Platform to ensure quality and readiness for
    products are loaded into the Data Access Center services,
    release date. The service involves human scientist/operator/programmer
    initial configurations of the payload, to monitor and analyze
    necessary, to make “hot fixes” during execution that maintain
    resulting data products.
    The contents of this document are subject to configuration control by the
    Team.
    22

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    7.1.3 Level 2 Quality Control
    This collects information on Level 2 science payload execution,
    data products from the Data Backbone to generate additional
    the measurement values against defined thresholds, providing
    capability for potentially detecting issues with the data
    telescope, camera, or data acquisition. Alarms stemming
    ered to LSST Data Facility Production Scientists for verification,
    7.2 Interfaces
    Calibration Products Production Execution and Data Release
    Backbone: for large-scale productions, a workflow system
    lected database entries from the Data Backbone to local
    payloads via the Data Butler. Similarly, the staging system
    alogs into the Data Backbone.
    8 Satellite Computing Enclave
    8.1 Service Description
    8.1.1 Data Release Production Satellite Processing
    This service controls the processing of jobs on the CC-IN2P3
    der the overall workload and workflow management of the Data
    tion service at NCSA.
    8.2 Interfaces
    DRP Satellite Processing to DRP Execution: DRP Execution
    cessing facility. All raw data, including images, metadata,
    transferred. Intermediate data products will be transferred
    Domain.
    The contents of this document are subject to configuration control by the
    Team.
    23

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    9 Data Access Center Enclaves
    There are two Data Access Centers, one in the US at NCSA
    These DACs are responsible for all science-user-facing
    LSST Science Platform (LSP). The LSP is the preferred analytic
    in the DAC. It provides computation and data access on both
    timescales. The US DAC also includes a service for distributing
    nual (Data Release) timescales to partner institutions,
    Public Outreach (EPO).
    The services in this domain must support multiple users simultaneously
    must be responsive to science user needs; updates are likely
    from the other domains as a result. The LSP must operate reliably
    is not impeded.
    9.1 Service Descriptions
    9.1.1 Bulk Data Distribution
    This service is used to transmit Level 1 and Level 2 data products
    ucation and Public Outreach, the UK LSST project, and the
    It extracts data products from the Data Backbone and transmits
    connections to designated, pre-subscribed partners.
    9.1.2 LSST Science Platform DAC instances
    This service provides an exploratory analysis environment
    ther broken down into three “Aspects” that it presents to
    “backend services” that users can take advantage
    7.
    of, as illustrated
    The “Portal” Aspect provides a pre-specified yet flexible discovery,
    “JupyterLab” Aspect provides a fully flexible (“notebook”)
    ing of images, catalogs, and plots and providing for execution
    algorithms. The “Web API” Aspect provides a language-independent,
    vices data access API with extensions for LSST capabilities
    The contents of this document are subject to configuration control by the
    Team.
    24

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Platform Backend Services
    LSST Science Platform
    LSP Web
    APIs
    Aspect
    LSP Portal
    Aspect
    LSP
    JupyterLab
    Aspect
    User
    Compute
    User File
    Storage
    User
    Databases
    Figure
    LSST
    7:
    Science Platform
    via all three Aspects to all data products, including images,
    API Aspect regenerates “virtual” data products on demand
    The backend services provide general-purpose user computation,
    sion; user file storage accessible to all three Aspects; and
    tables. Data may be shared with individual users, with groups,
    rights holders). Resource management of the backend services
    quota with additional resources allocated by a committee.
    All usage of any LSST Science Platform instance requires
    only to LSST data rights holders or LSST operations staff.
    9.2 Interfaces
    Bulk Data Distribution and LSST Science Platform to Data
    vices retrieve their data, including raw images, nightly
    products, metadata, and provenance, from the Data Backbone.
    the LSP Web APIs to retrieve data. The LSP JupyterLab Aspect
    also can use the Data Butler client library to access the
    Bulk Data Distribution to partners: The exact delivery mechanism
    bution is TBD.
    The contents of this document are subject to configuration control by the
    Team.
    25

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    10 NCSA Enclave Analysis and Developer Support
    This domain encompasses environments for analysts, developers,
    Its users are the Observatory staff as they analyze raw data
    characterize them, develop new algorithms and systems,
    nents and services before deployment.
    10.1 Service Descriptions
    10.1.1 LSST Science Platform Science Validation instance
    This instance of the LSST Science Platform is customized
    intermediate data products from the Alert, Calibration
    tions. It is optimized for usage by scientists within the
    lected external scientists can be granted access to assist
    optimization is to size and configure the three Aspects of
    more JupyterLab usage and less portal usage is expected.
    10.1.2 Developer Services
    Software version control service, packaging, build and unit
    agement, ticket tracking service, documentation services,
    10.1.3 Integration and Testing
    Integration environments representing various deployment
    vices, test datasets, test execution services, metric measurement
    This service includes the DM instance of the Camera DAQ Test
    integration environment, as well as the Prototype Data Access
    Access Center Enclave integration environment.
    The contents of this document are subject to configuration control by the
    Team.
    26

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    10.2 Interfaces
    LSP Science Validation instance and Integration and Testing
    vices in this domain interface with the Data Backbone. The
    is used to inspect, analyze, and validate the data products
    prior to their release and so has access to those products
    be used to annotate the data products, it can also write to
    tion and Testing environment may use raw data, intermediate
    products to perform tests.
    Developer Services do not have direct interfaces with the
    communicate via the distributed source version control
    system, and the configuration system. There is a separate
    Data Backbone to manage and track datasets used during development.
    11 Commissioning Cluster
    11.1 Service Description
    11.1.1 LSST Science Platform Commissioning instance
    This instance of the LSST Science Platform for Science Validation
    Cluster at the Base Facility (but also has access to computational
    accesses a Base endpoint for the Data Backbone. This location
    of both access to Data Backbone-resident data (which does
    the international network) and, perhaps more importantly,
    staff in Chile, which are served locally. Note that the Commissioning
    direct access to the Camera Data System; it relies on the
    Commissioning Cluster will have direct access to the OCS’s
    transformation).
    The contents of this document are subject to configuration control by the
    Team.
    27

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    11.2 Interfaces
    Commissioning Cluster to Data Backbone: The Commissioning
    bone for its data, like the other instances of the LSST Science
    Commissioning Cluster to EFD: The Commissioning Cluster
    to the Base replica of the EFD (before transformation).
    12 Infrastructure Domain
    This domain encompasses the underlying services and systems
    vironments in which the other domains are deployed and operate.
    domains but has no direct users.
    12.1 Service Descriptions
    12.1.1 Data Backbone
    The Data Backbone is a key component that provides for data
    cation, allowing data products to move between computational
    provides policy-based replication of files (in the Science
    Science Catalog Archive) across multiple physical locations,
    ing Cluster, NCSA, and DACs. It manages caches of files at
    tence to long-term archival storage (e.g. tape). It provides
    datasets and database entries and a retrieval mechanism
    The Qserv distributed database system for large-scale catalog
    Data Backbone in each DAC as well as in the NCSA Enclave.
    The relationships between the Data Backbone8.
    components
    The contents of this document are subject to configuration control by the
    Team.
    28

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Data
    Backbone
    Endpoint
    Tape
    Data Butler
    Client
    Science Image Archive
    Science Catalog Archive
    Data Backbone Transport and Replication
    Metadata &
    Provenance
    Files
    Qserv
    Other DBs
    Caches
    Science
    Image
    Archive
    Science
    Catalog
    Archive
    Science
    Image
    Archive
    Science
    Catalog
    Archive
    Data
    Backbone
    Endpoint
    Caches
    Other
    Backbone
    Clients
    Data Butler
    Client
    Other
    Backbone
    Clients
    Figure
    Data
    8:
    Backbone
    The contents of this document are subject to configuration control by the
    Team.
    29

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    12.1.2 Management/Monitoring
    These services provide management and monitoring at service
    levels for each enclave and domain.
    12.1.3 Provisioning/Deployment
    These services provide compute, local-to-node storage,
    for all processing, including Prompt Processing, Batch
    They allow allocation of compute and storage resources as
    deployment of services onto those resources.
    Some compute resources are reserved for particular uses,
    sioned, up to a certain maximum quota, if needed to deal with
    The priority order for processing is:
    •Prompt processing
    •Offlineprocessing
    •OCS-controlled batch processing
    •LSP Commissioning Cluster processing
    •LSP Science Validation processing
    •LSP Data Access Center processing
    •Template and Calibration Products Production
    •Data Release Production
    The Base Enclave’s services are not highly dynamic or flexible,
    terfacing to the OCS and Camera Data System. The baseline
    vSphere; they will be deployed using Puppet.
    The contents of this document are subject to configuration control by the
    Team.
    30

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    12.1.4 Workload/Workflow
    These services provide management of the execution of science
    gle pipeline to a series of “campaigns”, each consisting
    handle massively distributed computing, executing jobs
    and tracking their status and outputs. They ensure that the
    to it and that outputs (including log files, if any) are preserved.
    multiple computing environments, in particular between
    Facility at CC-IN2P3.
    12.1.5 Batch Processing
    This service provides execution of batch jobs with a variety
    in a variety of environments (e.g. OS and software configurations)
    sioned compute resources. It will use containerization
    ments. HTCondor is the baseline technology choice for this
    12.1.6 Identity Management
    This service provides authentication and authorization
    especially the LSST Science Platform instances.
    12.2 Interfaces
    The infrastructure services generally interact with all
    Enclave, there is no workload/workflow management or batch
    deployment are also simplified.
    Identity management instances are present in the Base and
    will be maintained at the Summit.) These are used to support
    tion for the other physically co-located enclaves: the Commissioning
    Access Centers.
    The contents of this document are subject to configuration control by the
    Team.
    31

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    13 Software Components
    13.1 Science Payloads
    These payloads are described in more detail inLDM-
    the DM Applications
    151
    ). Payloads are built from application software components.
    13.1.1 Alert Production Payload
    Executes under control of the Prompt Processing service.
    products including Alerts (with the exception of Solar System
    into the Data Backbone and Level 1 Database. Transmits Alerts
    Generates image quality feedback to the OCS and observers
    crosstalk-corrected science images and associated metadata
    cessing service; uses Master Calibration Images, Template
    ibration Database information from the Data Backbone.
    13.1.2 MOPS Payload
    Executes under control of the Offline Processing service after
    plete. Generates entries in the MOPS Database and the Level
    System Object records, measurements, and orbits. Performs
    of transients. Uses Level 1 Database entries and images from
    13.1.3 Raw Calibration Validation Payload
    Executes under control of the Prompt Processing service.
    quality feedback to the OCS and observers via the Telemetry
    science images and associated metadata delivered by the
    Calibration Images, and Calibration Database information
    The contents of this document are subject to configuration control by the
    Team.
    32

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    13.1.4 Daily Calibration Products Update Payload
    Executes under control of the OCS-controlled batch processing
    can be synchronized with the observing schedule. Uses raw
    mation from the Transformed EFD to generate a subset of
    Calibration Database entries in the Data Backbone.
    13.1.5 Periodic Calibration Products Production Payload
    Executes under control of the Template and CPP Execution
    tervals but perhaps as frequently as weekly or as infrequently
    stability of Observatory systems and their calibrations.
    formation from the Transformed EFD to generate a subset of
    Calibration Database entries in the Data Backbone.
    13.1.6 Template Generation Payload
    Executes under control of the Template and CPP Execution
    templates for Alert Production in between annual Data Release
    images to generate the templates, placing them in the Data
    13.1.7 Annual Calibration Products Production Payload
    Executes under control of the Template and CPP Execution
    to the start of the Data Release Production. Uses raw calibration
    the Transformed EFD, information from the Auxiliary Telescope
    catalogs to generate Master Calibration Images and Calibration
    Backbone.
    13.1.8 Data Release Production Payload
    Executes under control of the DRP Execution service at annual
    DRP” over a small portion of the sky, followed by the full
    science data products in the Data Backbone.
    The contents of this document are subject to configuration control by the
    Team.
    33

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    13.2 SUIT
    The Science User Interface and Tools provide visualization,
    ing, and searching elements that can be assembled into predetermined
    be used flexibly within dynamic “notebook” environments.
    13.3 Middleware
    13.3.1 Data Butler Access Client
    The Data Butler provides an access abstraction for all science
    underlying data sources and destinations to be configured
    ends ranging from local disk to network locations and a variety
    YAML and FITS files (extensible to HDF5 or ASDF) to database
    available within the LSST Science Platform JupyterLab environment.
    13.3.2 Parallel Distributed Database (Qserv)
    Underlying the catalog data access web service is a parallel
    handle the petabyte-scale, tens-of-trillions-of-rows
    13.3.3 Task Framework
    The Task Framework is a Python class library that provides
    entry points and conventions) to organize low-level algorithms
    rithmic components (Tasks; e.g. dark frame subtraction,
    ment), and to organize tasks into basic pipelines (SuperTasks;
    a coadd, difference a visit). The algorithmic code is written
    classes and providing implementation for standard entry
    the pipelines to be constructed and run at the level of a
    synchronized nodes. It allows for sub-node parallelization:
    cution, as well as providing (in the future) parallelization
    core Tasks and synchronized multi-node Tasks.
    The Task Framework serves as an interface layer between orchestration
    The contents of this document are subject to configuration control by the
    Team.
    34

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    code. It exposes a standard interface to “activators” (command-line
    orchestration layer and QA systems), which use it to execute
    Task Framework does not concern itself with fault-tolerant
    pipelines over multiple (thousands) of nodes nor any staging
    this is the concern of the orchestration middleware.
    The Task Framework exposes to the orchestration system needs
    derlying algorithmic code (i.e., the number of cores needed,
    pected need for data). It may also receive from the orchestration
    how to optimally run the particular task (i.e., which level
    desired).
    It also includes a configuration API and a logging API.
    14 Design and Implementation Standards
    Standards have been adopted by the DM Change Control Board
    nent designs within the LSST DM System. Coding standards
    to design may be found in the LSST
    1]).
    DM Developer Guide ([
    14.1 Python Version
    All LSST DM code will run under Python 3 by 2018-01-31. Packa
    have their code backward-compatible with Python 2.7, but
    Pipelines code that is
    lsst_apps
    dependency
    , and then only
    of
    until external dependencies
    become Python 3-only.
    Python 2.7 will be deprecated by the time LSST begins operations,
    dencies are in the process of moving to being Python 3-only.
    14.2 HTTPS Protocol
    In the absence of a specific technical justification and acceptance
    Security Officer and DM Change Control Board, all Web-enabled
    The contents of this document are subject to configuration control by the
    Team.
    35

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    vices exposed to users and the public Internet will use the
    protocol. To reiterate: this is only a default, and exceptions
    The covered interfaces include those of the three LSP Aspects
    APIs).
    The requirement to implement data access policies limiting
    holders will require all, or nearly all, data access to be
    nical justification. In addition, it appears to be appropriate
    current Internet environment, in the absence of good reasons
    The contents of this document are subject to configuration control by the
    Team.
    36

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    15 Appendix: Traceability
    15.1 Requirement to Component Traceability
    Note that DMS-REQ-0006 has no components; this requirement
    dropped.
    Requirement
    Components
    DMS-REQ-0059 Bad Pixel Map
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0060 Bias Residual
    OCSImage
    Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0061 Crosstalk Correction Matrix
    OCS Driven Batch, Calibration
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0062
    Illumination
    Correction
    Frame
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0063 Monochromatic Flatfield Data
    Cube
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0130 Calibration Data Products
    OCS Driven Batch, Calibration
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0132 Calibration Image Prove-
    nance
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0282 Dark Current Correction
    Frame
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0283 Fringe Correction
    OCS Driven
    Frame
    Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0018 Raw Science Image Data Ac-
    quisition
    Image and EFD Archiving
    The contents of this document are subject to configuration control by the
    Team.
    37

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0020 Wavefront Sensor Data Ac-
    quisition
    Image and EFD Archiving
    DMS-REQ-0022 Crosstalk Corrected Science
    Image Data Acquisition
    Prompt Processing
    DMS-REQ-0024 Raw Image Assembly
    Image and EFD Archiving
    DMS-REQ-0068 Raw Science Image Metadata Image and EFD Archiving
    DMS-REQ-0265 Guider Calibration Data Ac-
    quisition
    Image and EFD Archiving, OCS Driven Batch,
    ucts Production Execution, Annual Calibration,
    tion Update, Periodic Calibration
    DMS-REQ-0326 Storing Approximations of
    Per-pixel Metadata
    Data Release Production
    DMS-REQ-0331 Computing Derived Quanti-
    ties
    LSP Web APIs, Data Release Production
    DMS-REQ-0332 Denormalizing Database Ta-
    bles
    LSP Web APIs
    DMS-REQ-0333 Maximum Likelihood Values
    and Covariances
    Data Access Client, Alert Production, Data
    DMS-REQ-0346 Data Availability
    Image and EFD Archiving, OCS Driven Batch,
    ing, Prompt Processing, Calibration Products
    cution, Data Release Production Execution
    DMS-REQ-0347 Measurements in catalogs
    Data Access Client, LSP Web
    lease Production
    DMS-REQ-0004 Nightly Data Accessible
    Within 24 hrs
    Alert Distribution, Alert Filtering, Offline
    Processing, Alert Production, MOPS and Forced
    DMS-REQ-0010 Difference Exposures
    Alert Production
    DMS-REQ-0074 Difference Exposure At-
    tributes
    Alert Production
    DMS-REQ-0069 Processed Visit
    AlertImages
    Production
    DMS-REQ-0029 Generate Photometric Zero-
    point for Visit Image
    Alert Production
    DMS-REQ-0030 Generate WCS for Visit Im-
    ages
    Alert Production
    DMS-REQ-0070 Generate PSF for Visit Images Alert Production
    DMS-REQ-0072 Processed Visit Image Con-
    tent
    Alert Production
    DMS-REQ-0327 Background Model Calcula-
    tion
    Alert Production
    DMS-REQ-0328 Documenting Image Charac-
    terization
    Alert Production
    The contents of this document are subject to configuration control by the
    Team.
    38

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0097 Level 1 Data Quality Report
    Definition
    QC System, Alert Production
    DMS-REQ-0099 Level 1 Performance Report
    Definition
    Prompt Processing, QC System
    DMS-REQ-0101 Level 1 Calibration Report
    Definition
    OCS Driven Batch, Daily Calibration Update,
    DMS-REQ-0266 Exposure Catalog
    Alert Production
    DMS-REQ-0269 DIASource Catalog
    Alert Production
    DMS-REQ-0270 Faint DIASource Measure-
    ments
    Alert Production
    DMS-REQ-0271 DIAObject Catalog
    Alert Production
    DMS-REQ-0272 DIAObject Attributes
    Alert Production
    DMS-REQ-0273 SSObject Catalog
    MOPS and Forced Photometry
    DMS-REQ-0274 Alert Content
    Alert Production
    DMS-REQ-0317 DIAForcedSource
    Alert Production,
    Catalog
    MOPS and Forced Photometry
    DMS-REQ-0319 Characterizing
    AlertVariability
    Production, MOPS and Forced Photometry
    DMS-REQ-0323 Calculating SSObject Parame-
    ters
    LSP Web APIs
    DMS-REQ-0324 Matching DIASources to Ob-
    jects
    LSP Web APIs, Alert Production
    DMS-REQ-0325 Regenerating L1 Data Prod-
    ucts During Data Release Processing
    Data Release Production
    DMS-REQ-0034 Associate Sources to Objects Data Release Production
    DMS-REQ-0047 Provide PSF for Coadded Im-
    ages
    Data Release Production
    DMS-REQ-0103 Produce Images
    Datafor
    Release
    EPO
    Production
    DMS-REQ-0106 Coadded Image Provenance
    Data Release Production
    DMS-REQ-0267 Source Catalog
    Data Release Production
    DMS-REQ-0268 Forced-Source
    DataCatalog
    Release Production
    DMS-REQ-0275 Object Catalog
    Data Release Production
    DMS-REQ-0046 Provide Photometric Red-
    shifts of Galaxies
    Data Release Production
    DMS-REQ-0276 Object Characterization
    Data Release Production
    DMS-REQ-0277 Coadd Source
    Data
    Catalog
    Release Production
    DMS-REQ-0349 Detecting extended low sur-
    face brightness objects
    Data Release Production
    DMS-REQ-0278 Coadd Image Method Con-
    straints
    Data Release Production
    The contents of this document are subject to configuration control by the
    Team.
    39

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0279 Deep Detection
    DataCoadds
    Release Production
    DMS-REQ-0280 Template Coadds
    Data Release Production, Template Generation
    DMS-REQ-0281 Multi-bandData
    Coadds
    Release Production
    DMS-REQ-0329 All-Sky Visualization of Data
    Releases
    Data Release Production
    DMS-REQ-0330 Best SeeingData
    Coadds
    Release Production
    DMS-REQ-0334 Persisting DataBackbone,
    Products LSP Web APIs
    DMS-REQ-0335 PSF-Matched
    Data
    Coadds
    Release Production
    DMS-REQ-0336 Regenerating Data Products
    from Previous Data Releases
    Data Backbone, LSP Web APIs
    DMS-REQ-0337 Detecting faint variable ob-
    jects
    Data Release Production
    DMS-REQ-0338 Targeted Coadds
    Data Backbone, LSP Web APIs
    DMS-REQ-0339 Tracking Characterization
    Changes Between Data Releases
    Data Backbone, LSP Web APIs
    DMS-REQ-0320 Processing of Data From Spe-
    cial Programs
    Task Execution Framework, Data Release Production
    DMS-REQ-0321 Level 1 Processing of Special
    Programs Data
    OfflineProcessing,PromptProcessing,Alert
    and Forced Photometry
    DMS-REQ-0322 Special Programs Database
    Data Backbone, LSP Web APIs
    DMS-REQ-0344 Constraints on Level 1 Special
    Program Products Generation
    Offline Processing, Prompt Processing, Data
    Web APIs, Alert Production, MOPS and Forced
    DMS-REQ-0185 Archive Center
    Bulk Distribution, Data Backbone, ITC Environments
    DMS-REQ-0186 Archive Center Disaster Re-
    covery
    Bulk Distribution, Data Backbone, ITC Environments
    DMS-REQ-0187 Archive Center Co-Location
    with Existing Facility
    ITC Environments
    DMS-REQ-0188 Archive to Data Access Center
    Network
    Networks
    DMS-REQ-0189 Archive to Data Access Center
    Network Availability
    Networks
    DMS-REQ-0190 Archive to Data Access Center
    Network Reliability
    Networks
    DMS-REQ-0191 Archive to Data Access Center
    Network Secondary Link
    Networks
    DMS-REQ-0176 Base Facility Infrastructure
    Data Backbone, ITC Environments
    DMS-REQ-0177 Base Facility Temporary Stor-
    age
    Data Backbone
    The contents of this document are subject to configuration control by the
    Team.
    40

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0178 Base Facility Co-Location with
    Existing Facility
    ITC Environments
    DMS-REQ-0316 Commissioning
    ITC Environments
    Cluster
    DMS-REQ-0180 Base to Archive
    Networks
    DMS-REQ-0181 Base to Archive Network
    Availability
    Networks
    DMS-REQ-0182 Base to Archive Network Reli-
    ability
    Networks
    DMS-REQ-0183 Base to Archive Network Sec-
    ondary Link
    Networks
    DMS-REQ-0008 Pipeline Availability
    Alert Distribution, Alert Filtering, Image
    Driven Batch, Offline Processing, Prompt Processing,
    try Gateway, Calibration Products Production
    Release Production Execution, Data Backbone,
    ments, Networks
    DMS-REQ-0161 Optimization of Cost, Reliabil-
    ity and Availability in Order
    Alert Distribution, Alert Filtering, Data
    ments, Networks, LSP Web APIs, LSP JupyterLab,
    DMS-REQ-0162 Pipeline Throughput
    Alert Distribution, Alert Filtering, Image
    OCS Driven Batch, Offline Processing, Prompt
    Backbone, ITC Environments, Networks
    DMS-REQ-0163 Re-processing
    Calibration
    CapacityProducts Production Execution,
    duction Execution, Data Backbone, ITC Environments,
    works
    DMS-REQ-0164 Temporary Storage for Com-
    munications Links
    Data Backbone
    DMS-REQ-0165 Infrastructure Sizing for
    “catching up”
    Image and EFD Archiving, Offline Processing,
    ITC Environments, Networks
    DMS-REQ-0166 Incorporate Fault-Tolerance
    Data Backbone
    DMS-REQ-0167 Incorporate
    Alert
    Autonomics
    Distribution, Alert Filtering, Image
    Driven Batch, Offline Processing, Prompt Processing,
    tion Products Production Execution, Data
    Execution, Data Backbone, ITC Environments,
    DMS-REQ-0314 Compute Platform Hetero-
    geneity
    Alert Distribution, Alert Filtering, Image
    OCS Driven Batch, Offline Processing, Prompt
    Telemetry Gateway, Calibration Products
    tion, Data Release Production Execution,
    Proposal Manager, Developer Services, Integration
    Data Backbone, Identity Manager, ITC Environments,
    works, Data Access Client, Task Execution Framework,
    APIs, LSP JupyterLab, LSP Portal, QC System
    The contents of this document are subject to configuration control by the
    Team.
    41

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0318 Data Management Unsched-
    uled Downtime
    Alert Distribution, Alert Filtering, Image
    OCS Driven Batch, Offline Processing, Prompt
    Telemetry Gateway, Calibration Products
    tion, Data Release Production Execution,
    Proposal Manager, Data Backbone, Identity
    ronments, Networks, LSP Web APIs, LSP JupyterLab,
    QC System
    DMS-REQ-0122 Access to catalogs for exter-
    nal Level 3 processing
    Bulk Distribution, Data Backbone
    DMS-REQ-0123 Access to input catalogs for
    DAC-based Level 3 processing
    Data Backbone, LSP Web APIs, LSP JupyterLab,
    DMS-REQ-0124 Federation with external cat-
    alogs
    Data Backbone, LSP Web APIs, LSP JupyterLab,
    DMS-REQ-0126 Access to images for external
    Level 3 processing
    Bulk Distribution, Data Backbone
    DMS-REQ-0127 Access to input images for
    DAC-based Level 3 processing
    Data Backbone, LSP Web APIs, LSP JupyterLab,
    DMS-REQ-0193 Data AccessBulk
    Centers
    Distribution, Data Backbone, ITC Environments,
    works, LSP Web APIs, LSP JupyterLab, LSP Portal
    DMS-REQ-0194 Data Access Center Simulta-
    neous Connections
    Networks, LSP JupyterLab, LSP Portal
    DMS-REQ-0196 Data Access Center Geo-
    graphical Distribution
    ITC Environments
    DMS-REQ-0197 No Limit on Data Access Cen-
    ters
    Data Backbone, Identity Manager, ITC Environments,
    works, LSP Web APIs, LSP JupyterLab, LSP Portal
    DMS-REQ-0075 Catalog Queries
    LSP Web APIs
    DMS-REQ-0077 Maintain Archive Publicly Ac-
    cessible
    Data Backbone
    DMS-REQ-0078 Catalog Export
    LSP Web
    Formats
    APIs
    DMS-REQ-0094 Keep Historical Alert Archive
    Data Backbone
    DMS-REQ-0102 Provide Engineering and Fa-
    cility Database Archive
    Image and EFD Archiving, Data Backbone
    DMS-REQ-0309 Raw Data Archiving Reliability Image and EFD Archiving,
    DMS-REQ-0310 Un-Archived Data Product
    Cache
    Data Backbone
    DMS-REQ-0311Regenerate Un-archived Data
    Products
    Data Backbone, LSP Web APIs
    DMS-REQ-0312 Level 1 Data Product Access
    Prompt Processing, Data
    duction
    The contents of this document are subject to configuration control by the
    Team.
    42

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0313 Level 1 and 2 Catalog Access
    Data Backbone, LSP Web APIs
    DMS-REQ-0341 Providing a Precovery Service Offline Processing, LSP
    DMS-REQ-0345 Logging of catalog queries
    LSP Web APIs
    DMS-REQ-0168 Summit Facility Data Commu-
    nications
    Networks
    DMS-REQ-0170 Prefer Computing and Stor-
    age Down
    ITC Environments
    DMS-REQ-0315 DMS Communication with
    OCS
    Image and EFD Archiving, OCS Driven Batch,
    ing, Telemetry Gateway
    DMS-REQ-0171 Summit to Base
    Networks
    DMS-REQ-0172 Summit to Base Network
    Availability
    Networks
    DMS-REQ-0173 Summit to Base Network Re-
    liability
    Networks
    DMS-REQ-0174 Summit to Base Network Sec-
    ondary Link
    Networks
    DMS-REQ-0175 Summit to Base Network
    Ownership and Operation
    Networks
    DMS-REQ-0002 Transient Alert Distribution
    Alert Distribution, Alert
    duction
    DMS-REQ-0089 Solar System Objects Avail-
    able Within Specified Time
    Offline Processing, Data Backbone, LSP Web APIs,
    Forced Photometry
    DMS-REQ-0096 Generate Data Quality Report
    Within Specified Time
    QC System
    DMS-REQ-0098 Generate DMS Performance
    Report Within Specified Time
    QC System
    DMS-REQ-0100 Generate Calibration Report
    Within Specified Time
    QC System, Daily Calibration Update
    DMS-REQ-0131 Calibration Images Available
    Within Specified Time
    OCS Driven Batch, Prompt Processing, Daily
    date, Raw Calibration
    DMS-REQ-0284 Level-1 Production Complete-
    ness
    Image and EFD Archiving, Offline Processing,
    ing
    DMS-REQ-0285 Level 1 Source Association
    Alert Production
    DMS-REQ-0286 SSObject Precovery
    MOPS and Forced Photometry
    DMS-REQ-0287 DIASource Precovery
    Data Backbone, LSP Web APIs, MOPS and Forced
    DMS-REQ-0288UseofExternalOrbitCatalogs Alert Production, MOPS
    DMS-REQ-0342 Alert Filtering
    AlertService
    Filtering
    DMS-REQ-0343 Performance Requirements
    for LSST Alert Filtering Service
    Alert Distribution, Alert Filtering
    The contents of this document are subject to configuration control by the
    Team.
    43

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0348 Pre-definedAlert
    alertFiltering
    filters
    DMS-REQ-0289 Calibration Production Pro-
    cessing
    OCS Driven Batch, Calibration Products Production
    Annual Calibration, Daily Calibration Update,
    tion
    DMS-REQ-0006 Timely Publication of Level 2
    Data Releases
    DMS-REQ-0350 Associating Objects across
    data releases
    Data Backbone, Data Release Production
    DMS-REQ-0291 Query Repeatibility
    Data Backbone, LSP Web APIs
    DMS-REQ-0292 Uniqueness of IDs Across
    Data Releases
    Data Backbone, LSP Web APIs
    DMS-REQ-0293 Selection of Datasets
    Backbone, LSP Web APIs
    DMS-REQ-0294 ProcessingOCS
    of Datasets
    Driven Batch, Offline Processing, Prompt
    ibration Products Production Execution, Data
    tion Execution, Data Backbone
    DMS-REQ-0295 Transparent
    LSP
    Data
    WebAccess
    APIs
    DMS-REQ-0119 DAC resource allocation for
    Level 3 processing
    Bulk Distribution, Proposal Manager, LSP
    JupyterLab, LSP Portal
    DMS-REQ-0120 Level 3 Data Product Self Con-
    sistency
    Data Backbone, Workload+Workflow, LSP Web APIs
    DMS-REQ-0121 Provenance for Level 3 pro-
    cessing at DACs
    Data Backbone, Workload+Workflow, Data Access
    Execution Framework, LSP Web APIs
    DMS-REQ-0125 Software framework for Level
    3 catalog processing
    Data Backbone, Workload+Workflow, Data Access
    Execution Framework, LSP Web APIs
    DMS-REQ-0128 Software framework for Level
    3 image processing
    Data Backbone, Workload+Workflow, Data Access
    Execution Framework, LSP Web APIs
    DMS-REQ-0290 Level 3
    Data
    Import
    Backbone, LSP Web APIs
    DMS-REQ-0340 Access Controls of Level 3
    Data Products
    Data Backbone, Identity Manager
    DMS-REQ-0009 Simulated Data
    Alert Production, Annual Calibration, Daily
    Data Release Production, MOPS and Forced Photometry,
    odic Calibration, Raw Calibration, Template
    DMS-REQ-0032 Image Differencing
    Science Algorithms, Alert Production, Data
    DMS-REQ-0033 Provide Source Detection
    Software
    Science Algorithms, Alert Production, Data
    DMS-REQ-0042 Provide Astrometric Model
    ScienceAlgorithms,Alert
    Science Primitives
    DMS-REQ-0043 Provide Calibrated Photome-
    try
    Science Algorithms, Alert Production, Data
    Science Primitives
    The contents of this document are subject to configuration control by the
    Team.
    44

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Requirement
    Components
    DMS-REQ-0052 Enable a Range of Shape
    Measurement Approaches
    Science Algorithms, Alert Production, Data
    Science Primitives
    DMS-REQ-0160 Provide User Interface Ser-
    vices
    LSP Portal
    DMS-REQ-0296 Pre-cursor,
    Data
    andBackbone,
    Real Data Data Access Client, Science
    DMS-REQ-0351 Provide Beam Projector Coor-
    dinate Calculation Software
    Science Primitives
    DMS-REQ-0308 Software Architecture to En-
    able Community Re-Use
    Science Algorithms, Alert Production, Annual
    Calibration Update, Data Release Production,
    Forced Photometry, Periodic Calibration,
    Template Generation
    DMS-REQ-0065 Provide Image Access Ser-
    vices
    LSP Web APIs
    DMS-REQ-0155 Provide Data Access Services LSP Web APIs
    DMS-REQ-0298 Data Product and Raw Data
    Access
    Data Backbone, LSP Web APIs
    DMS-REQ-0299 Data Product
    Data
    Ingest
    Backbone, LSP Web APIs
    DMS-REQ-0300 Bulk Download
    Bulk
    Service
    Distribution, Data Backbone
    DMS-REQ-0156 Provide Pipeline Execution
    Services
    Workload+Workflow
    DMS-REQ-0302 ProductionWorkload+Workflow
    Orchestration
    DMS-REQ-0303 ProductionWorkload+Workflow
    Monitoring
    DMS-REQ-0304 Production Fault Tolerance
    Workload+Workflow, Task
    DMS-REQ-0158 Provide Pipeline Construction
    Services
    Task Execution Framework
    DMS-REQ-0305 Task Specification
    Task Execution Framework
    DMS-REQ-0306 Task Configuration
    Task Execution Framework
    DMS-REQ-0297 DMS Initialization Compo-
    nent
    ITC Environments
    DMS-REQ-0301 Control of Level-1 Production Prompt Processing
    DMS-REQ-0307 Unique Processing Coverage Workload+Workflow
    15.2 Component to Requirement Traceability
    Note that only “leaf” components are traced to requirements.
    Component
    Requirements
    Data Management
    The contents of this document are subject to configuration control by the
    Team.
    45

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Level 1 System
    Alert Distribution
    •DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0161 Optimization of Cost, Reliability and
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0002 Transient Alert Distribution
    •DMS-REQ-0343 Performance Requirements for LSST
    vice
    Alert Filtering
    •DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0161 Optimization of Cost, Reliability and
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0002 Transient Alert Distribution
    •DMS-REQ-0342 Alert Filtering Service
    •DMS-REQ-0343 Performance Requirements for LSST
    vice
    •DMS-REQ-0348 Pre-defined alert filters
    Image and EFD Archiving
    •DMS-REQ-0018 Raw Science Image Data Acquisition
    •DMS-REQ-0020 Wavefront Sensor Data Acquisition
    •DMS-REQ-0024 Raw Image Assembly
    •DMS-REQ-0068 Raw Science Image Metadata
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0165 Infrastructure Sizing for “catching
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0102 Provide Engineering and Facility Database
    •DMS-REQ-0309 Raw Data Archiving Reliability
    •DMS-REQ-0315 DMS Communication with OCS
    •DMS-REQ-0284 Level-1 Production Completeness
    The contents of this document are subject to configuration control by the
    Team.
    46

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    OCS Driven Batch
    •DMS-REQ-0059 Bad Pixel Map
    •DMS-REQ-0060 Bias Residual Image
    •DMS-REQ-0061 Crosstalk Correction Matrix
    •DMS-REQ-0062 Illumination Correction Frame
    •DMS-REQ-0063 Monochromatic Flatfield Data Cube
    •DMS-REQ-0130 Calibration Data Products
    •DMS-REQ-0132 Calibration Image Provenance
    •DMS-REQ-0282 Dark Current Correction Frame
    •DMS-REQ-0283 Fringe Correction Frame
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0101 Level 1 Calibration Report Definition
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0315 DMS Communication with OCS
    •DMS-REQ-0131 Calibration Images Available Within Specified
    •DMS-REQ-0289 Calibration Production Processing
    •DMS-REQ-0294 Processing of Datasets
    The contents of this document are subject to configuration control by the
    Team.
    47

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Offline Processing
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
    •DMS-REQ-0321 Level 1 Processing of Special Programs
    •DMS-REQ-0344 Constraints on Level 1 Special Program
    ation
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0165 Infrastructure Sizing for “catching up”
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0341 Providing a Precovery Service
    •DMS-REQ-0089 Solar System Objects Available Within
    •DMS-REQ-0284 Level-1 Production Completeness
    •DMS-REQ-0294 Processing of Datasets
    Prompt Processing
    •DMS-REQ-0022 Crosstalk Corrected Science Image Data
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
    •DMS-REQ-0099 Level 1 Performance Report Definition
    •DMS-REQ-0321 Level 1 Processing of Special Programs
    •DMS-REQ-0344 Constraints on Level 1 Special Program
    ation
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0312 Level 1 Data Product Access
    •DMS-REQ-0315 DMS Communication with OCS
    •DMS-REQ-0002 Transient Alert Distribution
    •DMS-REQ-0131 Calibration Images Available Within Specified
    •DMS-REQ-0284 Level-1 Production Completeness
    •DMS-REQ-0294 Processing of Datasets
    •DMS-REQ-0301 Control of Level-1 Production
    The contents of this document are subject to configuration control by the
    Team.
    48

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Telemetry Gateway
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled
    •DMS-REQ-0315 DMS Communication with OCS
    Level 2 System
    Calibration Products Production Execution
    •DMS-REQ-0059 Bad Pixel Map
    •DMS-REQ-0060 Bias Residual Image
    •DMS-REQ-0061 Crosstalk Correction Matrix
    •DMS-REQ-0062 Illumination Correction Frame
    •DMS-REQ-0063 Monochromatic Flatfield Data
    •DMS-REQ-0130 Calibration Data Products
    •DMS-REQ-0132 Calibration Image Provenance
    •DMS-REQ-0282 Dark Current Correction Frame
    •DMS-REQ-0283 Fringe Correction Frame
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0163 Re-processing Capacity
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled
    •DMS-REQ-0289 Calibration Production Processing
    •DMS-REQ-0294 Processing of Datasets
    The contents of this document are subject to configuration control by the
    Team.
    49

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Data Release Production Execution
    •DMS-REQ-0346 Data Availability
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0163 Re-processing Capacity
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0294 Processing of Datasets
    Data Access Center
    Bulk Distribution
    •DMS-REQ-0185 Archive Center
    •DMS-REQ-0186 Archive Center Disaster Recovery
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0122 Access to catalogs for external
    •DMS-REQ-0126 Access to images for external Level
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0119 DAC resource allocation for Level
    •DMS-REQ-0300 Bulk Download Service
    Proposal Manager
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0119 DAC resource allocation for Level
    Analysis and Developer Services
    Developer Services
    •DMS-REQ-0314 Compute Platform Heterogeneity
    Integration and Test
    •DMS-REQ-0314 Compute Platform Heterogeneity
    Infrastructure
    Data Backbone
    •DMS-REQ-0334 Persisting Data Products
    •DMS-REQ-0336RegeneratingDataProductsfrom
    •DMS-REQ-0338 Targeted Coadds
    •DMS-REQ-0339 Tracking Characterization Changes
    leases
    •DMS-REQ-0322 Special Programs Database
    •DMS-REQ-0344 Constraints on Level 1 Special
    ation
    •DMS-REQ-0185 Archive Center
    •DMS-REQ-0186 Archive Center Disaster Recovery
    •DMS-REQ-0176 Base Facility Infrastructure
    •DMS-REQ-0177 Base Facility Temporary Storage
    The contents of this document are subject to configuration control by the
    Team.
    50

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Data Backbone (continued)
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0161 Optimization of Cost, Reliability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0163 Re-processing Capacity
    •DMS-REQ-0164 Temporary Storage for Communications
    •DMS-REQ-0165 Infrastructure Sizing for “catching
    •DMS-REQ-0166 Incorporate Fault-Tolerance
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0122 Access to catalogs for external Level
    •DMS-REQ-0123 Access to input catalogs for DAC-based
    •DMS-REQ-0124 Federation with external catalogs
    •DMS-REQ-0126 Access to images for external Level
    •DMS-REQ-0127 Access to input images for DAC-based
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0077 Maintain Archive Publicly Accessible
    •DMS-REQ-0094 Keep Historical Alert Archive
    •DMS-REQ-0102 Provide Engineering and Facility Database
    •DMS-REQ-0309 Raw Data Archiving Reliability
    •DMS-REQ-0310 Un-Archived Data Product Cache
    •DMS-REQ-0311 Regenerate Un-archived Data Products
    •DMS-REQ-0312 Level 1 Data Product Access
    •DMS-REQ-0313 Level 1 and 2 Catalog Access
    •DMS-REQ-0089 Solar System Objects Available Within
    •DMS-REQ-0287 DIASource Precovery
    •DMS-REQ-0350 Associating Objects across data releases
    •DMS-REQ-0291 Query Repeatibility
    •DMS-REQ-0292 Uniqueness of IDs Across Data Releases
    •DMS-REQ-0293 Selection of Datasets
    •DMS-REQ-0294 Processing of Datasets
    •DMS-REQ-0120 Level 3 Data Product Self Consistency
    •DMS-REQ-0121 Provenance for Level 3 processing at
    •DMS-REQ-0125 Software framework for Level 3 catalog
    •DMS-REQ-0128 Software framework for Level 3 image
    •DMS-REQ-0290 Level 3 Data Import
    •DMS-REQ-0340 Access Controls of Level 3 Data Products
    •DMS-REQ-0296 Pre-cursor, and Real Data
    •DMS-REQ-0298 Data Product and Raw Data Access
    The contents of this document are subject to configuration control by the
    Team.
    51

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Data Backbone (continued)
    •DMS-REQ-0299 Data Product Ingest
    •DMS-REQ-0300 Bulk Download Service
    Identity Manager
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0340 Access Controls of Level 3 Data Products
    ITC Environments
    •DMS-REQ-0185 Archive Center
    •DMS-REQ-0186 Archive Center Disaster Recovery
    •DMS-REQ-0187 Archive Center Co-Location with Existing
    •DMS-REQ-0176 Base Facility Infrastructure
    •DMS-REQ-0178 Base Facility Co-Location with Existing
    •DMS-REQ-0316 Commissioning Cluster
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0161 Optimization of Cost, Reliability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0163 Re-processing Capacity
    •DMS-REQ-0165 Infrastructure Sizing for “catching
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0196 Data Access Center Geographical Distribution
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0170 Prefer Computing and Storage Down
    •DMS-REQ-0297 DMS Initialization Component
    The contents of this document are subject to configuration control by the
    Team.
    52

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Networks
    •DMS-REQ-0188 Archive to Data Access Center Network
    •DMS-REQ-0189 Archive to Data Access Center Network Availability
    •DMS-REQ-0190 Archive to Data Access Center Network Reliability
    •DMS-REQ-0191 Archive to Data Access Center Network Secondary
    •DMS-REQ-0180 Base to Archive Network
    •DMS-REQ-0181 Base to Archive Network Availability
    •DMS-REQ-0182 Base to Archive Network Reliability
    •DMS-REQ-0183 Base to Archive Network Secondary Link
    •DMS-REQ-0008 Pipeline Availability
    •DMS-REQ-0161 Optimization of Cost, Reliability and Availability
    •DMS-REQ-0162 Pipeline Throughput
    •DMS-REQ-0163 Re-processing Capacity
    •DMS-REQ-0165 Infrastructure Sizing for “catching up”
    •DMS-REQ-0167 Incorporate Autonomics
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0194 Data Access Center Simultaneous Connections
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0168 Summit Facility Data Communications
    •DMS-REQ-0171 Summit to Base Network
    •DMS-REQ-0172 Summit to Base Network Availability
    •DMS-REQ-0173 Summit to Base Network Reliability
    •DMS-REQ-0174 Summit to Base Network Secondary Link
    •DMS-REQ-0175 Summit to Base Network Ownership and Operation
    The contents of this document are subject to configuration control by the
    Team.
    53

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Workload+Workflow
    •DMS-REQ-0120 Level 3 Data Product Self Consistency
    •DMS-REQ-0121 Provenance for Level 3 processing at
    •DMS-REQ-0125 Software framework for Level 3 catalog
    •DMS-REQ-0128 Software framework for Level 3 image
    •DMS-REQ-0156 Provide Pipeline Execution Services
    •DMS-REQ-0302 Production Orchestration
    •DMS-REQ-0303 Production Monitoring
    •DMS-REQ-0304 Production Fault Tolerance
    Data Access Client
    •DMS-REQ-0333 Maximum Likelihood Values and Covariances
    •DMS-REQ-0347 Measurements in catalogs
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0121 Provenance for Level 3 processing at
    •DMS-REQ-0125 Software framework for Level 3 catalog
    •DMS-REQ-0128 Software framework for Level 3 image
    •DMS-REQ-0296 Pre-cursor, and Real Data
    Task Execution Framework
    •DMS-REQ-0320 Processing of Data From Special Programs
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0121 Provenance for Level 3 processing at
    •DMS-REQ-0125 Software framework for Level 3 catalog
    •DMS-REQ-0128 Software framework for Level 3 image
    •DMS-REQ-0304 Production Fault Tolerance
    •DMS-REQ-0158 Provide Pipeline Construction Services
    •DMS-REQ-0305 Task Specification
    •DMS-REQ-0306 Task Configuration
    Science Platform
    LSP Web APIs
    •DMS-REQ-0331 Computing Derived Quantities
    •DMS-REQ-0332 Denormalizing Database Tables
    •DMS-REQ-0347 Measurements in catalogs
    •DMS-REQ-0323 Calculating SSObject Parameters
    •DMS-REQ-0324 Matching DIASources to Objects
    •DMS-REQ-0334 Persisting Data Products
    •DMS-REQ-0336RegeneratingDataProductsfromPrevious
    •DMS-REQ-0338 Targeted Coadds
    •DMS-REQ-0339 Tracking Characterization Changes
    leases
    •DMS-REQ-0322 Special Programs Database
    •DMS-REQ-0344 Constraints on Level 1 Special Program
    ation
    •DMS-REQ-0161 Optimization of Cost, Reliability
    The contents of this document are subject to configuration control by the
    Team.
    54

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    LSP Web APIs (continued)
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0123 Access to input catalogs for DAC-based
    •DMS-REQ-0124 Federation with external catalogs
    •DMS-REQ-0127 Access to input images for DAC-based
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0075 Catalog Queries
    •DMS-REQ-0078 Catalog Export Formats
    •DMS-REQ-0311 Regenerate Un-archived Data Products
    •DMS-REQ-0312 Level 1 Data Product Access
    •DMS-REQ-0313 Level 1 and 2 Catalog Access
    •DMS-REQ-0345 Logging of catalog queries
    •DMS-REQ-0089 Solar System Objects Available Within
    •DMS-REQ-0287 DIASource Precovery
    •DMS-REQ-0291 Query Repeatibility
    •DMS-REQ-0292 Uniqueness of IDs Across Data Releases
    •DMS-REQ-0293 Selection of Datasets
    •DMS-REQ-0295 Transparent Data Access
    •DMS-REQ-0119 DAC resource allocation for Level 3
    •DMS-REQ-0120 Level 3 Data Product Self Consistency
    •DMS-REQ-0121 Provenance for Level 3 processing at
    •DMS-REQ-0125 Software framework for Level 3 catalog
    •DMS-REQ-0128 Software framework for Level 3 image
    •DMS-REQ-0290 Level 3 Data Import
    •DMS-REQ-0065 Provide Image Access Services
    •DMS-REQ-0155 Provide Data Access Services
    •DMS-REQ-0298 Data Product and Raw Data Access
    •DMS-REQ-0299 Data Product Ingest
    The contents of this document are subject to configuration control by the
    Team.
    55

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    LSP JupyterLab
    •DMS-REQ-0161 Optimization of Cost, Reliability and Availability
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0123 Access to input catalogs for DAC-based Level
    •DMS-REQ-0124 Federation with external catalogs
    •DMS-REQ-0127 Access to input images for DAC-based Level
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0194 Data Access Center Simultaneous Connections
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0119 DAC resource allocation for Level 3 processing
    LSP Portal
    •DMS-REQ-0161 Optimization of Cost, Reliability and Availability
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0123 Access to input catalogs for DAC-based Level
    •DMS-REQ-0124 Federation with external catalogs
    •DMS-REQ-0127 Access to input images for DAC-based Level
    •DMS-REQ-0193 Data Access Centers
    •DMS-REQ-0194 Data Access Center Simultaneous Connections
    •DMS-REQ-0197 No Limit on Data Access Centers
    •DMS-REQ-0341 Providing a Precovery Service
    •DMS-REQ-0119 DAC resource allocation for Level 3 processing
    •DMS-REQ-0160 Provide User Interface Services
    QC System
    •DMS-REQ-0097 Level 1 Data Quality Report Definition
    •DMS-REQ-0099 Level 1 Performance Report Definition
    •DMS-REQ-0314 Compute Platform Heterogeneity
    •DMS-REQ-0318 Data Management Unscheduled Downtime
    •DMS-REQ-0096 Generate Data Quality Report Within Specified
    •DMS-REQ-0098 Generate DMS Performance Report Within
    Time
    •DMS-REQ-0100 Generate Calibration Report Within Specified
    The contents of this document are subject to configuration control by the
    Team.
    56

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Science Algorithms
    •DMS-REQ-0032 Image Differencing
    •DMS-REQ-0033 Provide Source Detection Software
    •DMS-REQ-0042 Provide Astrometric Model
    •DMS-REQ-0043 Provide Calibrated Photometry
    •DMS-REQ-0052 Enable a Range of Shape Measurement Approaches
    •DMS-REQ-0296 Pre-cursor, and Real Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    Science Payloads
    Alert Production
    •DMS-REQ-0333 Maximum Likelihood Values and Covariances
    •DMS-REQ-0347 Measurements in catalogs
    •DMS-REQ-0004 Nightly Data Accessible Within 24 hrs
    •DMS-REQ-0010 Difference Exposures
    •DMS-REQ-0074 Difference Exposure Attributes
    •DMS-REQ-0069 Processed Visit Images
    •DMS-REQ-0029 Generate Photometric Zeropoint for Visit
    •DMS-REQ-0030 Generate WCS for Visit Images
    •DMS-REQ-0070 Generate PSF for Visit Images
    •DMS-REQ-0072 Processed Visit Image Content
    •DMS-REQ-0327 Background Model Calculation
    •DMS-REQ-0328 Documenting Image Characterization
    •DMS-REQ-0097 Level 1 Data Quality Report Definition
    •DMS-REQ-0266 Exposure Catalog
    •DMS-REQ-0269 DIASource Catalog
    •DMS-REQ-0270 Faint DIASource Measurements
    •DMS-REQ-0271 DIAObject Catalog
    •DMS-REQ-0272 DIAObject Attributes
    •DMS-REQ-0274 Alert Content
    •DMS-REQ-0317 DIAForcedSource Catalog
    •DMS-REQ-0319 Characterizing Variability
    •DMS-REQ-0324 Matching DIASources to Objects
    •DMS-REQ-0321 Level 1 Processing of Special Programs
    The contents of this document are subject to configuration control by the
    Team.
    57

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Alert Production (continued)
    •DMS-REQ-0344 Constraints on Level 1 Special Program
    ation
    •DMS-REQ-0312 Level 1 Data Product Access
    •DMS-REQ-0002 Transient Alert Distribution
    •DMS-REQ-0285 Level 1 Source Association
    •DMS-REQ-0288 Use of External Orbit Catalogs
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0032 Image Differencing
    •DMS-REQ-0033 Provide Source Detection Software
    •DMS-REQ-0042 Provide Astrometric Model
    •DMS-REQ-0043 Provide Calibrated Photometry
    •DMS-REQ-0052 Enable a Range of Shape Measurement
    •DMS-REQ-0308 Software Architecture to Enable Community
    Annual Calibration
    •DMS-REQ-0059 Bad Pixel Map
    •DMS-REQ-0060 Bias Residual Image
    •DMS-REQ-0061 Crosstalk Correction Matrix
    •DMS-REQ-0062 Illumination Correction Frame
    •DMS-REQ-0063 Monochromatic Flatfield Data Cube
    •DMS-REQ-0130 Calibration Data Products
    •DMS-REQ-0132 Calibration Image Provenance
    •DMS-REQ-0282 Dark Current Correction Frame
    •DMS-REQ-0283 Fringe Correction Frame
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0289 Calibration Production Processing
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    The contents of this document are subject to configuration control by the
    Team.
    58

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Daily Calibration Update
    •DMS-REQ-0059 Bad Pixel Map
    •DMS-REQ-0060 Bias Residual Image
    •DMS-REQ-0061 Crosstalk Correction Matrix
    •DMS-REQ-0062 Illumination Correction Frame
    •DMS-REQ-0063 Monochromatic Flatfield Data Cube
    •DMS-REQ-0130 Calibration Data Products
    •DMS-REQ-0132 Calibration Image Provenance
    •DMS-REQ-0282 Dark Current Correction Frame
    •DMS-REQ-0283 Fringe Correction Frame
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0101 Level 1 Calibration Report Definition
    •DMS-REQ-0100 Generate Calibration Report Within
    •DMS-REQ-0131 Calibration Images Available Within
    •DMS-REQ-0289 Calibration Production Processing
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    Data Release Production
    •DMS-REQ-0326 Storing Approximations of Per-pixel
    •DMS-REQ-0331 Computing Derived Quantities
    •DMS-REQ-0333 Maximum Likelihood Values and Covariances
    •DMS-REQ-0347 Measurements in catalogs
    •DMS-REQ-0325 Regenerating L1 Data Products During
    cessing
    •DMS-REQ-0034 Associate Sources to Objects
    •DMS-REQ-0047 Provide PSF for Coadded Images
    •DMS-REQ-0103 Produce Images for EPO
    •DMS-REQ-0106 Coadded Image Provenance
    •DMS-REQ-0267 Source Catalog
    •DMS-REQ-0268 Forced-Source Catalog
    •DMS-REQ-0275 Object Catalog
    •DMS-REQ-0046 Provide Photometric Redshifts of Galaxies
    •DMS-REQ-0276 Object Characterization
    •DMS-REQ-0277 Coadd Source Catalog
    •DMS-REQ-0349 Detecting extended low surface brightness
    •DMS-REQ-0278 Coadd Image Method Constraints
    •DMS-REQ-0279 Deep Detection Coadds
    •DMS-REQ-0280 Template Coadds
    •DMS-REQ-0281 Multi-band Coadds
    •DMS-REQ-0329 All-Sky Visualization of Data Releases
    •DMS-REQ-0330 Best Seeing Coadds
    •DMS-REQ-0335 PSF-Matched Coadds
    •DMS-REQ-0337 Detecting faint variable objects
    The contents of this document are subject to configuration control by the
    Team.
    59

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Data Release Production (continued)
    •DMS-REQ-0320 Processing of Data From Special
    •DMS-REQ-0350 Associating Objects across data
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0032 Image Differencing
    •DMS-REQ-0033 Provide Source Detection Software
    •DMS-REQ-0042 Provide Astrometric Model
    •DMS-REQ-0043 Provide Calibrated Photometry
    •DMS-REQ-0052 Enable a Range of Shape Measurement
    •DMS-REQ-0308 Software Architecture to Enable
    MOPS and Forced Photometry
    •DMS-REQ-0004 Nightly Data Accessible Within
    •DMS-REQ-0273 SSObject Catalog
    •DMS-REQ-0317 DIAForcedSource Catalog
    •DMS-REQ-0319 Characterizing Variability
    •DMS-REQ-0321 Level 1 Processing of Special
    •DMS-REQ-0344 Constraints on Level 1 Special
    ation
    •DMS-REQ-0341 Providing a Precovery Service
    •DMS-REQ-0089 Solar System Objects Available
    •DMS-REQ-0286 SSObject Precovery
    •DMS-REQ-0287 DIASource Precovery
    •DMS-REQ-0288 Use of External Orbit Catalogs
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable
    The contents of this document are subject to configuration control by the
    Team.
    60

    LARGE SYNOPTIC SURVEY TELESCOPE
    Data Management System DesignLDM-148
    Latest Revision 2017-07-04
    Component
    Requirements
    Periodic Calibration
    •DMS-REQ-0059 Bad Pixel Map
    •DMS-REQ-0060 Bias Residual Image
    •DMS-REQ-0061 Crosstalk Correction Matrix
    •DMS-REQ-0062 Illumination Correction Frame
    •DMS-REQ-0063 Monochromatic Flatfield Data Cube
    •DMS-REQ-0130 Calibration Data Products
    •DMS-REQ-0132 Calibration Image Provenance
    •DMS-REQ-0282 Dark Current Correction Frame
    •DMS-REQ-0283 Fringe Correction Frame
    •DMS-REQ-0265 Guider Calibration Data Acquisition
    •DMS-REQ-0289 Calibration Production Processing
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    Raw Calibration
    •DMS-REQ-0101 Level 1 Calibration Report Definition
    •DMS-REQ-0131 Calibration Images Available Within Specified
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    Template Generation
    •DMS-REQ-0280 Template Coadds
    •DMS-REQ-0009 Simulated Data
    •DMS-REQ-0308 Software Architecture to Enable Community
    Science Primitives
    •DMS-REQ-0042 Provide Astrometric Model
    •DMS-REQ-0043 Provide Calibrated Photometry
    •DMS-REQ-0052 Enable a Range of Shape Measurement Approaches
    •DMS-REQ-0351 Provide Beam Projector Coordinate Calculation
    16 References
    [1]
    LSST DM
    Developer
    https://developer.lsst.io/
    Guide, URL
    [2]
    [LDM-139]
    , Becla, J., Lim,
    Data
    K.T.,
    Management
    2013,
    Storage Sizing
    ,LDM-139,
    and I/O Model Explanation
    URLhttps://ls.st/LDM-139
    [3][LDM-141],Becla,J.,Lim,K.T.,2013,DataManagementStorageSizingandI/OModel,LDM-141,URLhttps:
    //ls.st/LDM-141
    Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
    Team.
    61

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    [4][LDM-135],Becla,J.,Wang,D.,Monkewitz,S.,etal.,2013,DatabaseDesign,LDM-135,URLhttps://ls.st/
    LDM-135
    [5][LSE-29],Claver,C.F.,TheLSSTSystemsEngineeringIntegratedProjectLSSTSystemRequirements,
    LSE-29,URLhttps://ls.st/LSE-29
    [6][LSE-30],Claver,C.F.,TheLSSTSystemsEngineeringIntegratedProjectLSSTSystemRequirements,
    LSE-30,URLhttps://ls.st/LSE-30
    [7][LSE-72],Dubious-Felsmann,G.,Schumacher,G.,Selvy,B.,2014,OCSCommandDictionaryforData
    ment,LSE-72,URLhttps://ls.st/LSE-72
    [8][LSE-81],Dubois-Felsmann,G.,2013,LSSTScienceandProjectSizingInputs,LSE-81,URLhttps://ls.st/LSE-81
    [9][LSE-61],Dubois-Felsmann,G.,2016,LSSTDataManagementSubsystemRequirements,LSE-61,URLhttps:
    //ls.st/LSE-61
    [10][LSE-82],Dubois-Felsmann,G.,Lim,K.T.,2013,ScienceandProjectSizingInputsExplanation,LSE-82,URL
    https://ls.st/LSE-82
    [11][LDM-542],Dubois-Felsmann,G.,Lim,K.T.,Wu,X.,etal.,2017,LSSTSciencePlatformDesign,LDM-542,URL
    https://ls.st/LDM-542
    [12][LDM-129],Freemon,M.,Kantor,J.,2013,DataManagementInfrastructureDesign,LDM-129,URLhttps:
    //ls.st/LDM-129
    [13][LDM-143],Freemon,M.,Pietrowicz,S.,2013,SiteSpecificInfrastructureEstimation,LDM-143,
    URLhttps://ls.st/LDM-143
    [14][LDM-144],Freemon,M.,Pietrowicz,S.,Alt,J.,2016,SiteSpecificInfrastructureEstimation,LDM-144,
    URLhttps://ls.st/LDM-144
    [15][LPM-17],Ivezić,Ž.,TheLSSTScienceCollaboration,2011,LSSTScienceRequirementsDocument,LPM-17,URL
    https://ls.st/LPM-17
    [16][LSE-163],Jurić,M.,etal.,2017,LSSTDataProductsDefinitionDocument,LSE-163,URLhttps://ls.st/LSE-163
    [17][LDM-142],Kantor,J.,2017,NetworkSizingModel,LDM-142,URLhttps://ls.st/LDM-142
    [18][LDM-138],Kantor,J.,Axelrod,T.,Lim,K.T.,2013,DataManagementComputeSizingModel,LDM-138,URL
    https://ls.st/LDM-138
    [19][LSE-78],Lambert,R.,Kantor,J.,Huffer,M.,etal.,2017,LSSTObservatoryNetworkDesign,LSE-78,URLhttps:
    //ls.st/LSE-78
    [20][LDM-230],Lim,K.T.,2013,AutomatedOperationoftheLSSTDataManagementSystem,LDM-230,URLhttps:
    //ls.st/LDM-230
    [21][LDM-140],Lim,K.T.,Smith,C.,Axelrod,T.,Dubois-Felsmann,G.,Freemon,DataManagement
    ComputeSizingExplanation,LDM-140,URLhttps://ls.st/LDM-140
    Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
    Team.
    62

    LARGESYNOPTICSURVEYTELESCOPE
    DataManagementSystemDesignLDM-148LatestRevision2017-07-04
    [22][LDM-148],Lim,K.T.,Bosch,J.,Dubois-Felsmann,G.,etal.,2017,DataManagementSystemDesign,LDM-148,
    URLhttps://ls.st/LDM-148
    [23][LDM-152],Lim,K.T.,Dubois-Felsmann,G.,Johnson,M.,Jurić,M.,Petravick,DataManagement
    MiddlewareDesign,LDM-152,URLhttps://ls.st/LDM-152
    [24][Document-5373],Pinto,P.,Kantor,J.,Strauss,M.,Sweeney,D.,2008,DataAccessWhitePaper,Document-
    5373,URLhttps://ls.st/Document-5373
    [25][LDM-151],Swinbank,J.D.,etal.,2017,DataManagementSciencePipelinesDesign,LDM-151,URLhttps:
    //ls.st/LDM-151
    Thecontentsofthisdocumentaresubjecttoconfigurationcontrolbythe
    Team.
    63

    Back to top