Overview

The Manager is the operator control plane for the SAGRIS Open Data Cube. It is the single place to publish new Sentinel-1 imagery, keep the data layers consistent, and inspect the health of the platform. It does not reimplement the pipeline — it wraps the existing pipeline scripts (symlinks → metadata → index → catalog) behind a dashboard with a reconciliation engine, so every step is one click and every run is streamed live.

What this panel shows

The Overview is the landing page and a dashboard of the whole platform for the active profile (see Storage):

• Essential tasks — shortcut cards to the most common workflows.
• Pipeline stages — image counts — how many scenes exist at each stage of the chain (symlinks, previews, YAMLs, indexed). Gaps between stages tell you what still needs processing.
• Service health — a quick status line for the platform's services.

The publishing pipeline at a glance

Data moves through the platform in a fixed order. Each stage has its own panel:

1. Storage — choose which disks and region you are working on (the profile).
2. Symlinks — link newly-arrived GeoTIFFs into the regional tree.
3. Previews — check which scenes still need PNG previews.
4. Metadata — generate the EO3 YAML file for each scene.
5. Indexing — register the YAMLs in the ODC STAC index and refresh the catalog.

Tip: If you just want to publish a day's worth of fresh data, you don't need to run those five panels by hand — use Operations → Publish new data, which chains them into a single run.

Choosing the region

The region selector in the top bar (europe / africa) and the active profile picker on the Storage panel together scope everything you do. Set them first — almost every other panel operates only on the active profile's disks, years, and base folder.

Reading the top bar

Storage

The starting point for any session. This panel maps every NFS server and disk the platform can see, and lets you define profiles — the named selection of disks + region + base folder that scopes everything else you do in the Manager.

What this panel shows

The panel lists every disk on the configured NFS servers — including disks that aren't currently mounted — together with the number of symlinks pointing there. Disks are discovered live with showmount against each server, and the list always falls back to the canonical disk template, so it can never come up empty (the way it used to when nothing was mounted).

NFS servers to scan

The NFS servers to scan card lists the servers probed for shares. Add a server by name (e.g. x113, used in the /nfs/<name>/diskN paths) and an optional host (IP or hostname for showmount; defaults to the name). Save list & rescan persists it and re-probes. If the list is empty, it falls back to the SERVERS=() array in x101_links_v5.sh.

Profiles — the key concept

A profile ties together:

• a selection of disks (which storage to scan),
• a region (europe or africa),
• a base folder — the regional collection root that holds the incoming/ and processed/ queues and the YAML tree.

Every other panel (Symlinks, Metadata, Indexing, Operations…) operates on the active profile only. The symlink manager scans just the profile's disks and enqueues newly-linked scenes into <base_folder>/incoming/ so the metadata step can pick them up.

Set the active profile

1. Pick a profile from Active profile. Its disks, region, and base folder become the scope for the whole app.
2. The profile summary appears next to the picker so you can confirm you're on the right collection.

Create or edit a profile

1. Click Refresh inventory to list the disks (mounted or not).
2. Tick the disks this profile should cover. Disks already claimed by another profile are shown greyed-out and labelled “in profile …” — a disk can belong to only one profile, so they can't be ticked.
3. Click Save selection as new… to create a profile from the ticked disks, or edit the active profile inline.
4. In the editor set the Region, Base folder (e.g. /opt/sagris_europe), and an optional Description.
5. Click Save changes.

Note: The profile Name is read-only in the editor. To rename, use Save selection as new… — this preserves the old profile rather than overwriting it.

Unmounted disks can be selected — they'll be mounted when the pipeline runs. Only disks owned by another profile are locked.

Controls

Symlinks

Step 1 of the pipeline. This panel runs the symlink synchronisation (x101_links_v5.sh) for the active profile: it scans the NFS shares declared in the profile, refreshes or creates symlinks under the regional repository folder, and writes any newly-added scene names to <base_folder>/incoming/ so the Metadata step picks them up next.

Run this whenever new Sentinel-1 GeoTIFFs have landed on the NFS shares.

Step by step

1. Confirm the active profile is correct (set it on Storage). Its summary card appears at the top.
2. Leave Dry-run ticked for the first attempt — it counts what would change without touching the tree.
3. Under Years to process, pick one or more years. Use All, None, or Recent only as shortcuts. Each unchecked year is skipped entirely; symlinks already in place are left alone.
4. Click ⟳ Synchronise symlinks to run, or ⚡ Fast scan for a quick current-year pass (see below).
5. Watch the streamed output in the console. The progress bar and status line update live.
6. When the dry-run looks right, untick Dry-run and run again for real.

Fast scan vs full synchronise

Reading the results

• The console streams the script output line by line; ⏹ Stop halts a running job.
• Years are discovered from /opt/sagris/S1/GRD/images/gtif/.
• New scene base names are queued into <base_folder>/incoming/ for the metadata step.

Tip: Symlinks is the prerequisite for every later stage. The Previews, Metadata, and Indexing panels will refuse a year that has not been symlink-synced and send you back here with that year pre-selected.

Previews

Step 3 of the pipeline (read-only). This panel compares the symlinks in /opt/sagris/S1/GRD/images/gtif/ against the PNG previews and thumbnails in /opt/sagris/S1/GRD/images/pics/, and reports a year-by-month breakdown of how many scenes are still missing their pictures. Nothing on disk is modified.

Use it to find out exactly which calendar blocks an external preview-generation step needs to target.

Step by step

1. Confirm the active profile at the top.
2. Under Years to check, pick the years to scan (All / None / Recent only).
3. Click ⟳ Scan for missing pictures.
4. Read the per-month gap report that appears below the console.
5. Optionally click Export work order to save the list of missing scenes for the preview generator.

Reading the results

• The output is a grid of year × month counts of scenes that have no preview yet.
• The scan is read-only — it never creates or deletes pictures.
• Dry-run here only controls whether the run is recorded in the Manager's memory; the comparison itself always runs.

Year chip markers

Note: A year must be symlink-synced first (Symlinks). If it isn't, the preview scan is cancelled and the Symlinks tab opens with the missing year already checked.

Metadata (YAML generation)

Step 4 of the pipeline. This panel generates the EO3 YAML metadata file for each symlinked scene in the selected years (odc_s1_region_img_yaml.py). Each YAML references the symlink as its path and embeds the deterministic preview / thumbnail PNG URLs. The files land under <base_folder>/S1/GRD/images/yaml/{year}/{month}/.

A scene must have a YAML before it can be added to the index — this is what makes it a real ODC dataset.

Step by step

1. Confirm the active profile.
2. Keep Dry-run ticked first — the script counts the work without writing YAMLs.
3. Under Years to generate, choose the years (All / None / Recent only).
4. Optionally adjust workers (parallel generation, 1–16; default 4).
5. Click ⟳ Generate YAMLs, or click Verify consistency first for a fast, read-only comparison of symlinks vs existing YAMLs.
6. When the dry-run looks right, untick Dry-run and run for real.

Options

Reading the results

• The console streams progress per scene; the Verify consistency report shows the symlink-vs-YAML drift.
• Behind the scenes the Manager queues each selected year into <base_folder>/incoming/manager_yaml_<year>_*.txt, then moves it to processed/ on success.

Note: Every selected year must be symlink-synced first (Symlinks).

Indexing

Step 5 — the final pipeline stage. This panel reconciles the on-disk YAML repository for the active profile with the ODC STAC index (odc_s1_img_index.py) and refreshes the Explorer catalog so the data goes live.

The sync runs in three phases:

1. Inject — adds new YAMLs to the index, and with Force re-indexes updated ones.
2. Cleanup — archives index rows whose YAML has been deleted from disk (stale-index drift).
3. Refresh — runs cubedash-gen --force-refresh so the changes surface in the STAC catalog immediately.

Step by step

1. Confirm the active profile.
2. Click Verify consistency first — a read-only count of what's missing, what's stale, and what's already in sync.
3. Keep Dry-run ticked, choose the Years to index, then click ⟳ Run all 3 steps to preview the work.
4. When it looks right, untick Dry-run and run again for real.

Per-phase buttons

For diagnosis or to re-run a single phase, use the individual buttons instead of Run all 3 steps:

Options

Year chip markers

Note: Each selected year must have YAML history (Metadata) first; otherwise the run is cancelled and the Metadata tab opens with the years pre-selected.

Operations

Composite workflows that chain several pipeline panels into a single button, so you don't have to run Symlinks → Metadata → Indexing by hand. Each preset uses the active profile and is scoped to the years you select.

Choose the years first

The Years to process selector at the top applies to all presets below. Restricting to the year(s) you care about (current year by default) keeps a run from re-validating the entire multi-year tree — that's the difference between a quick pass and a long one.

The presets

Publish new data · ~5–15 min

The full ingestion chain for fresh imagery. Use it after new data lands on the NFS shares.

Chain: Symlinks (detect new) → YAML (process incoming) → Indexing (inject + refresh)

Keep Dry-run ticked to preview, then run for real.

Validate consistency · ~1–2 min · read-only

A health check across every data layer for the active profile. Reports drift at the storage, preview, YAML, and index levels. Safe to run any time — it writes nothing.

Checks: Storage inventory · Preview-sync · YAML consistency · Index consistency

Clean up orphans · ~3–10 min · deletes only

Periodic cleanup of accumulated cruft. Deletes only — never creates. Removes malformed/broken symlinks, orphan YAMLs (no matching symlink), and stale index rows (no YAML on disk), then refreshes Explorer.

Chain: Symlinks (delete broken) → YAML (orphan purge) → Indexing (archive stale + refresh)

Keep Dry-run ticked for the first attempt.

Restore / Ingest region · un-archive ~instant · ingest ~5–15 min

Brings a region's data back online. Un-archives any soft-deleted datasets (recovers an accidental mass-archive — dc.load and the data API see them again immediately), then refreshes Explorer/STAC. Tick Also ingest new TIFs to additionally run the full ingestion chain for unprocessed imagery. Use it after an accidental archive, or to publish a freshly-staged region (e.g. North Africa).

Chain: Un-archive → (optional) Symlinks → YAML → Index → Explorer/STAC refresh

Keep Dry-run ticked to preview the counts first. Un-archiving is fully reversible.

One run at a time

Only one workflow can run per profile at a time. Clicking a preset while another mutation is in progress for the same profile returns immediately (HTTP 409) rather than risking a conflicting change. Watch the shared console for streamed output; ⏹ Stop halts the run.

Downloads

Find Sentinel-1 acquisitions that exist on the Alaska Satellite Facility (ASF) archive but are missing locally, and export their download URLs as a work order. This panel is read-only — it discovers and compares, but never touches the ODC index or downloads anything itself.

The "processed" rule

A granule counts as processed only when both polarisations (VV and VH) are present as symlinks under /opt/sagris/S1/GRD/images/gtif/. If either is missing, the granule's download URL is added to the work-order list.

Step by step

1. Build a discovery batch: each row is one query scope — a saved region polygon (from Regions) plus a date range.
2. Click + Add region + period to add more rows.
3. Optionally tune the memory settings (see below).
4. Click ⟳ Discover & compare. Each ASF query auto-subdivides if it saturates; results are merged and diffed against the local tree.
5. When the run finishes, 💾 Save missing-URL list… becomes available — save it via the browser's Save As dialog and feed it to the downloader.

Memory layer

To avoid re-querying ASF for periods already confirmed complete:

Note: The discovery walks each row's range month-by-month. Months that only partially intersect a row's range are queried but not written to memory.

Reading the results

• The console streams the ASF discovery output.
• The outcome is the count of granules found, how many are already local (both-pol), and the missing-URL list ready to save.

Coverage

Check how evenly imagery is distributed over time for a region. The panel produces a daily image-count grid for the selected year — one column per month, January through the latest month — so you can compare time-series consistency month-over-month and spot supply gaps (for example, an ASF outage).

Step by step

1. In Countries (multi-select), pick one or more country boundaries to count over.
2. Choose the Polarisation (VV / VH / HH / HV) — one polarisation is counted per acquisition.
3. Set the Year.
4. Click RUN.
5. Read the daily-coverage grid below. The summary line reports the totals.

What the counts mean

• Each cell is the number of acquisitions on that day whose footprint intersects the selected country boundaries.
• Counting one polarisation per acquisition avoids double-counting VV+VH pairs.
• Sparse columns or empty stretches highlight gaps in the supply for that region.

Exporting

Regions

Build a region polygon (an area of interest) by selecting countries on a map. The selection is unioned, simplified, and saved as a region YAML that the Downloads panel can use immediately for ASF discovery.

Step by step

1. Under Build new region, click a country on the map to add it to the selection. Click again to remove it.
2. To pick a single piece of a country (an exclave or island such as Kaliningrad or Crimea), tick select individual parts first, then click the piece.
3. Adjust simplify tolerance (degrees) to trade detail for a smaller, faster polygon. The preview and stats update as you change it.
4. Give the region a name (and optional description).
5. Click 💾 Save region. Tick overwrite if exists if you are replacing one.

Existing regions

The Existing regions list at the top shows the regions already defined. Builder-made regions can be previewed on the map and edited (including refining an _asf outline's detail there) or deleted.

Note: Hand-drawn regions are protected — they appear in the list but are read-only here. The source YAML stays the source of truth; edit or delete those by touching the YAML files directly.

Controls

Maintenance & index operations

Direct ODC-index operations that sit outside the normal pipeline — for fixing index drift, archiving wrongly-indexed datasets, and restoring archived ones. Use this panel when the standard Indexing flow can't express what you need.

Why it's safe: Writes use a targeted SQL update paired with a cubedash force-refresh, and never go through ODC's add(), which would silently un-archive datasets.

Index health (read-only)

The top card is a read-only report of the current index state. Click ↻ Refresh to recompute it. Start here to understand what needs fixing before making any change.

Safe archive

Marks datasets in the index as archived (hidden from the catalog without deleting the files).

1. Choose the polarisation.
2. Optionally add an extra predicate to narrow the target — e.g. archive African footprints accidentally indexed into a Europe collection.
3. Click Plan archive to preview the generated SQL and the number of affected datasets — nothing is changed yet.
4. Review the impact line. When you're sure, tick commit and click Execute + force-refresh.

The mass-archive guard

The system refuses an archive that has no filter, or one that would touch more than 25% of active datasets. This guard exists because an over-broad archive previously took a whole product offline. Only tick allow mass-archive (override guard) when a large archive is genuinely intended.

Restore (un-archive)

The safe inverse of archive — brings archived datasets back into the catalog. Use it to recover from an archive that went too far.

Controls

Log Analyst

A read-only operations view over /opt/odcube/logs — service health, crash-loop detection, HTTP error rates, restart history, and pipeline run outcomes. It surfaces problems without you needing shell access to the server. Output is refreshed on demand (cached for up to 60 seconds) and access-log lines are redacted.

What each section shows

Service health

A status tile per platform service, with advisories below it flagging anything notable (a crash loop, a spike in errors, a recent restart). Click ↻ Refresh to re-scan.

Service detail

Pick a service from the dropdown to see its detail — recent activity, restart history, and error signal for that one service.

Pipeline runs

The outcomes of recent pipeline runs (the symlink / YAML / index jobs launched from the other panels), so you can confirm a publish actually succeeded.

Tail

View the most recent log lines for a chosen service:

1. Pick the service from the first dropdown.
2. Optionally filter by level (ERROR, CRITICAL, INFO, or all).
3. Click View.

How to use it

• Start at Service health — green across the board means nothing needs attention.
• If an advisory appears, open that service in Service detail, then use Tail filtered to ERROR to read the actual messages.
• After running a publish from Operations or Indexing, check Pipeline runs to confirm the outcome.

Note: This panel only reads logs — it never restarts services or changes anything. It's safe to open at any time.