---
title: "Getting started with seqout"
output:
  rmarkdown::html_vignette:
    self_contained: true
    mathjax: null
vignette: >
  %\VignetteIndexEntry{Getting started with seqout}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
seqout::SeqoutDefault(NULL)
# live API calls run only off CRAN, when online
online <- (identical(Sys.getenv("NOT_CRAN"), "true") ||
  identical(Sys.getenv("IN_PKGDOWN"), "true")) && seqout::SeqoutOnline()
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = online)
```

`seqout` is an R client for [seqout.org](https://seqout.org). It queries study-level and sample-level metadata from public genomic archives:
*   Gene Expression Omnibus (GEO)
*   Sequence Read Archive (SRA)
*   European Nucleotide Archive (ENA)
*   ArrayExpress
*   DNA Data Bank of Japan (DDBJ DRA and GEA)
*   Genome Sequence Archive (GSA)

## Installation

Install with `pak`:

```{r install, eval = FALSE}
# install.packages("pak")
pak::pak("saketlab/seqout/R")
```

The default backend is the REST API, which needs no API key or local setup.

```{r setup}
library(seqout)
```

## Search the archives

### Query studies

`SeqoutSearch()` searches all archives with free text, filters, or both.

```{r search}
SeqoutSearch("cervical cancer single cell", db = "geo", limit = 5)
```

### Query samples (cohorts)

`SampleSearch()` searches individual samples, filtering on harmonized metadata such as tissue, disease, and age across studies.

```{r cohort}
liver <- SampleSearch(
  organism = "Homo sapiens",
  tissue = "liver",
  age_min_years = 50,
  limit = 5
)
liver[, c("sample", "study_accession", "tissue", "disease", "sex", "age")]
```

For query operators and filters, see [Search](articles/search.html).

### Everything pipes

Each function takes its subject first (an accession, a `seqout_counts` table, a
list of cohort filters, or the connection when the call has no other subject),
so calls chain with R's native pipe.

```{r pipes}
# A PubMed ID or a DOI, told apart by the value, to the datasets that cite it
"34764296" |>
  Paper() |>
  nrow()

list(tissue = "liver", organism = "Homo sapiens") |>
  SampleSearch(limit = 3) |>
  subset(select = c(sample, study_accession, disease))
```

`con` is the last argument everywhere else.

## Retrieve study metadata

`SeqoutGet()` fetches metadata for a study, sample, or run accession and maps it to related records in other archives.

The returned dataset is lazy: each field is fetched from the API and cached on first read.

```{r dataset}
d <- SeqoutGet("GSE168652")
d
```

```{r dataset-fields}
d$meta$title
c(samples = nrow(d$samples), runs = nrow(d$runs), papers = nrow(d$pubs))
```

### Harmonized sample attributes

The `enriched` field holds harmonized sample-level and cell-level labels:

```{r enriched}
d$enriched[, c("sample", "tissue", "disease", "assay")]
```

For metadata fields, see [Metadata](articles/metadata.html).

## Read counts matrices

`seqout` resolves, downloads, and parses processed GEO supplementary files into R matrices.

### List counts files

`SeqoutListCounts()` lists the matrix units for a GEO accession without downloading any matrix files. A unit is the smallest file group that `seqout` can parse as one matrix, such as a 10x triplet, `.h5`, `.h5ad`, or `.rds` file.

```{r counts-table}
counts <- SeqoutListCounts("GSE297547")
m <- counts

nrow(m)
head(m[, c("unit", "sample", "format", "preferred", "has_metadata")], 4)
```

### Load a matrix

`SeqoutMatrix()` reads one unit into a list of `X`, `obs`, and `var`.

```{r counts-read, eval = FALSE}
rna <- SeqoutMatrix(counts, sample = "GSM8994520")
dim(rna$X)
```

### Convert to Seurat

`Seqout2Seurat()` accepts a matrix unit, GSM accession, GSE accession, or `SeqoutListCounts()` table. Extra arguments go to `CreateSeuratObject()`.

```{r counts-seurat, eval = FALSE}
obj <- Seqout2Seurat("GSE297547", sample = "GSM8994520", min.cells = 3, min.features = 200)
```

For unit selection and Seurat workflows, see [Counts](articles/counts.html).

### Convert old gene IDs and repair Excel errors

`SeqoutGeneNames()` uses genevintage to select an annotation and convert IDs.
`SeqoutCorrectNames()` repairs Excel-damaged symbols against a specified reference.
Both preserve expression values and keep an audit; `SeqoutGeneReport()` reads it.

```{r gene-names-demo, eval = FALSE}
# GSE52529: original Ensembl IDs, FPKM not raw counts
old <- SeqoutListCounts("GSE52529") |>
  SeqoutMatrix(sample = "GSE52529_fpkm_matrix.txt.gz")
named <- old |> SeqoutGeneNames()
named$gene_annotation[[1]]$annotation

# GSE101942: verified Excel-damaged gene names
damaged <- SeqoutListCounts("GSE101942") |>
  SeqoutMatrix(sample = "GSE101942_IPSC_rawCounts.txt.gz")
repaired <- damaged |> SeqoutCorrectNames(species = "human", release = 116)
report <- SeqoutGeneReport(repaired)
report[report$status != "unchanged", ]
```

genevintage selects GENCODE 17 on GRCh37 for GSE52529 without a version argument.
See [Gene names: old identifiers and Excel errors](gene-names.html) for runnable
excerpts and a comparison of historical and newer symbols.

## Download data files

Three functions download raw and processed files:
* `DownloadSupplementary()`: processed dataset files, such as counts matrices, annotations, and archives.
* `DownloadRuns()`: raw sequencing reads in FASTQ or SRA format.
* `DownloadBAMs()`: submitted BAM alignment files.

Each saves files in a directory named after the accession, verifies checksums, and skips files already on disk.

Check the `supplementary` field before downloading:

```{r supplementary}
head(d$supplementary, 4)
```

```{r downloads, eval = FALSE}
DownloadSupplementary("GSE168652", "GSE168652")
DownloadRuns("SRR12012336", "SRR12012336")
DownloadBAMs("ERP117016", "ERP117016")
```

For download modes and checksums, see [Downloads](articles/downloads.html).

## Choose a backend

| Backend | Mechanism | Best Use Cases |
| --- | --- | --- |
| `"api"` (default) | Queries the `seqout.org` REST API over HTTP. | Standard workflows. |
| `"parquet"` | Queries the published Parquet dump using DuckDB. | Offline workflows, large batch queries, and SQL analytics. |

Configure the Parquet backend explicitly to avoid accidental dump scans.

```{r parquet}
con <- SeqoutConnect("parquet", quiet = TRUE)

Query("SELECT count(*) AS n FROM geo_series", con = con)
```

Pass the connection through `con`, or make it the session default with `SeqoutDefault(con)`.

Close the Parquet connection with `SeqoutClose()`:

```{r close}
SeqoutClose(con)
```

For Parquet downloads, see [Downloads](articles/downloads.html).

## Next steps

* [Search](articles/search.html): query operators, filters, and sample cohort construction.
* [Metadata](articles/metadata.html): dataset structures, publication records, and read-derived metrics.
* [Counts](articles/counts.html): GEO supplementary file parsing and Seurat object construction.
* [Downloads](articles/downloads.html): raw reads, alignments, and database dumps.