Get started with nesstarR

.Nesstar is the file format of Nesstar Publisher. Nesstar Publisher is survey documentation software from the International Household Survey Network (IHSN). Statistical agencies, for example the Ministry of Statistics and Programme Implementation (MoSPI), India, publish survey microdata in this format. nesstarR reads these files in R. You do not need Nesstar software.

This vignette uses a small synthetic file that comes with the package. For an example with real survey data, refer to the article “Read NSS 68 data from the MoSPI portal” on the package website.

library(nesstarR)

Parse the file

nesstar_parse() decodes the resource index, the dataset descriptors and the variable directories. It does not decode row data, and it does not keep the file in memory. The other functions read only the bytes that they need from the file.

path <- system.file("extdata", "example.Nesstar", package = "nesstarR")
x <- nesstar_parse(path)
x
#> <nesstar_binary>
#>  File      : example.Nesstar 
#>  Datasets  : 1

List the datasets and variables

One .Nesstar file can contain many datasets. nesstar_datasets() gives the number of rows and variables in each dataset.

nesstar_datasets(x)
#>   dataset_number row_count variable_count
#> 1              1         4              4

nesstar_variables() gives the storage format of each variable in one dataset. Mode 5 is a numeric variable. Mode 1 is a string variable.

nesstar_variables(x, dataset_number = 1)
#>     name variable_id mode_code value_format_code value_offset_i64 width_value
#> 1    AGE         301         5                 3                0           0
#> 2  GROUP         302         5                 2               10           0
#> 3 INCOME         303         5                10                0           0
#> 4   NAME         304         1                 0                0           4
#>   data_offset data_length
#> 1         730           4
#> 2         734           2
#> 3         736          32
#> 4         768          16

Read a dataset

nesstar_read_dataset() decodes one dataset into a data frame. Missing values become NA.

nesstar_read_dataset(x, dataset_number = 1)
#>   AGE GROUP    INCOME NAME
#> 1  25    11    1500.5   ab
#> 2  40    12        NA  cde
#> 3  NA    NA       0.0     
#> 4  63    13 2000000.0 wxyz

To decode only some columns, use the columns argument.

nesstar_read_dataset(x, dataset_number = 1, columns = c("AGE", "NAME"))
#>   AGE NAME
#> 1  25   ab
#> 2  40  cde
#> 3  NA     
#> 4  63 wxyz

Read the labels

Many .Nesstar files contain XML blocks with variable labels and category codes. nesstar_metadata() decodes these blocks.

meta <- nesstar_metadata(x)
meta$datasets[[1]]$variables[[1]]
#> $name
#> [1] "AGE"
#> 
#> $label
#> [1] "Age in years"
#> 
#> $categories
#> list()

Some older files do not contain these blocks. For these files, nesstar_metadata() gives a warning and returns NULL.

Export to CSV

nesstar_export() writes each dataset to a different CSV file. By default, it reads 50,000 rows at a time and writes gzip files. Thus, the memory use stays low for large files.

out <- nesstar_export(x, output_dir = tempdir())
#> Wrote: example_ds01.csv.gz (4 rows)
basename(out)
#> [1] "example_ds01.csv.gz"
read.csv(out[1])
#>   AGE GROUP    INCOME NAME
#> 1  25    11    1500.5   ab
#> 2  40    12        NA  cde
#> 3  NA    NA       0.0     
#> 4  63    13 2000000.0 wxyz