ATN System Technical Overview:

The ATN DAC is an on line data distribution service for data retrieved from 
electronic tags attached to marine animals, as well as environmental data 
associated with those tag data. Data are supplied to the ATN DAC by animal 
tracking tag data researchers. The supplied data can be downloaded by anyone, 
within permission constraints set by the individual data providers. Some 
elements of data that are with withheld from direct download distribution may 
still be viewed graphically on line. Requests to access withheld data can be 
made via the ATN DAC download website.

User Contact:

   - In the early (beta) phases upload access to the ATN DAC will be more by
     invitation than user request - as time progresses, and more users want to
     include their data we will accept direct upload requests from the broader 
     tagging community. Requests can come via the contact form on the upload 
     page or by ATN admin contact email address as it is distributed over time.

Registration:

  - Registration is open to anyone in the tagging community. You only have to 
    register if you wish to make your own data available through the ATN DAC 
    system. Within permissions set by upload users, data downloads are available
    to anyone.

Authorization:

  - Registration on the ATN DAC is a multi-step process. After a user fills in
    their information on the registration page, their email address has been 
    confirmed as available, and their password accepted as strong enough, an 
    email message is automatically sent to the ATN DAC administration staff for
    confirmation and account setup. After a new user's account is available, an
    authorization code is sent to the email address provided in the user 
    registration request. 

Confirmation:

  - Once a user receives their authorization code and enters it on the 
    registration page, their account is automatically confirmed and their login
    credentials will immediately allow access to the ATN DAC data file upload
    and metadata form entry pages.

Login:

  - A validated user email and password login is required on the data file 
    upload page. The metadata entry form pages require entry of a validated user
    email but a password is not required. 

Uploads:

  - Once logged in on the data file upload page, a user may drag and drop data
    files onto the page, or they may use a file browser to select files for 
    upload. The page code assesses the submitted files and accepts or rejects
    based on file type. As accepted files are uploaded a progress bar will 
    become active beneath the drag and drop area. Uploaded files are sent to a
    user specific directory in the ATN DAC server file system. Once all 
    submitted files have been accepted and uploaded, or rejected, a new browser
    tab is opened presenting a list with the results. At the same time an email
    with the results is automatically sent to the ATN DAC administration system
    for further action, including returning an acknowledgement email to the user
    and submitting accepted data files to the proper datastream. Depending on 
    the submitted file types, these operations may be fully automatic or require 
    some manual preprocessing steps before handing off the automated systems.

Permissions:

  - There are three data download permission levels:

      (1) all data immediatlely available
      (2) ocean data immediately available, animal data embargoed for one year
      (3) all data embargoed for one year

    When data are submitted via the upload page, the permission level can be
    selected for each group of files uploaded. For data submitted via a tag
    vendor API system or other automated process, a default permission level
    is set for the incoming data during datastream setup. Those permission 
    levels may be changed later at the user's discretion. (Though, obviously,
    data initially specified as level (1) may be downloaded before their
    level is updated to (2) or (3).)

    Since the World Wide Web was not designed with data security in mind, 
    directly displaying protected track data with the unprotected track data 
    opens the protected data to retrieval by web savy individuals via browser
    development tools, even when data file downloads are blocked. The current
    solution to that problem is to display only the end point for a protected
    track on the main ATN DAC data display page, with indication that it is 
    protected data. Clicking that point will bring up an image of the protected
    track and give instructions on how to contact the data provider for access 
    to those data. This system may change over time.

Metadata:

  - A species and tag type dependent minimum set of metadata must be provided 
    before submitted tracking data can be accepted for inclusion on the ATN DAC
    system. Metadata may be submitted by (at least) three means: (1) email from
    a validated email address, (2) uploaded in files via the data upload page,
    or, (3) provided via the metadata upload forms pages. For a large number of
    tags submitted at one time, methods (1) and (2) may be preferrable over
    method (3).

Datastreams:

  - uploads:

      Data files submitted via the upload web page are directed to user specific
      directories on the ATN DAC servers. The permission level can be set for
      each group of files uploaded. Each group receives a unique upload session
      ID which, along with the individual user's unique ID, and the permission
      level, is prepened to upload file names. Automated routines scan the 
      user directories for new uploads. Depending on the type of file, different
      actions will be taken by the automated systems. For example, if a standard
      format SSM track file is submitted, that file will be automatically 
      processed and added to the ATN DAC data set. On the other hand, if a
      text file of unknown format is submitted, that file will be held for
      manual invesitgation. In all cases, an email is transmitted to the ATN
      DAC administrators and to the authorized user detailing how uploaded 
      files are being processed.     

  - APIs:

      The ATN DAC can accept data from the various tag vendor API systems. 
      Exaclty how those data are handled internally is API dependant. In all
      cases a default permission level is set for the datastream when it is
      established. Once an API datastream is established for a particular
      validated user, data delivery is fully automated.

  - client side processing:

      When a user wants the option, the ATN DAC staff can provide assistance in
      setting up services on the user's server that provide automatic access to 
      data from the user's server without passing data through a vendor API. 
      These may take the form of an ERDDAP server, automated distribution
      codes, e.g., curl, wget, or others, depending on each users' needs.

Automated Processing:

  - As much as possible, interactions with the ATN DAC system are direct 
    machine-to-machine transfers of data. Depending on how raw received data
    are, the entire process from submission to deployment on the ATN DAC web
    site may be fully automated, (e.g., a final format SSM track file), or, may 
    require some preprocessing steps before submission to the final automated
    update systems, (e.g., a raw PRV file).

NetCDF:

  - In general, data in place for distribution on the ATN DAC system are in the
    form of NetCDF files, which underly ERDDAP servers. Exceptions include 
    extremely large files, such as long term timeseries data, where these data
    are made available through pointers to their files in a postgreSQL database
    table that associates the files with the data provider.

Data As Filesystem Files:

  - When a large data set is made available as a file in the filesystem rather
    than through one of the NetCDF based ERDDAP servers, that data set file can
    be downloaded via the ATN DAC main web site interface. ATN DAC upload group
    users do not have direct access to the ATN DAC filesystem. The pointer to
    the file available in the postgreSQL database is used by the ATN DAC web
    page system to provide the file.

ERDDAP Access:

  - The ATN DAC web site does allow some user access to its own ERDDAP server.
    ERDDAP servers on the HMS side of the ATN system may or may not allow user
    access, depending on the nature of the data. E.g., data with permission 
    level set higher than 1 are in an password protected ERDDAP server and only
    available to ATN DAC administrators.

Legacy Data:

  - We are still sending some data out through the old TOPP nightly FTP system.
    We wish we weren't...;^)