# πŸŽπŸ’¨vroom The fastest delimited reader for R, **1.23 GB/sec**. ![Animated GIF of Taylor Swift in a car](https://raw.githubusercontent.com/tidyverse/vroom/main/img/taylor.gif) But that’s impossible! How can it be [so fast](https://vroom.tidyverse.org/articles/benchmarks.html)? vroom doesn’t stop to actually *read* all of your data, it simply indexes where each record is located so it can be read later. The vectors returned use the [Altrep framework](https://svn.r-project.org/R/branches/ALTREP/ALTREP.html) to lazily load the data on-demand when it is accessed, so you only pay for what you use. This lazy access is done automatically, so no changes to your R data-manipulation code are needed. vroom also uses multiple threads for indexing, materializing non-character columns, and when writing to further improve performance. | package | version | time (sec) | speedup | throughput | |:-----------|--------:|-----------:|--------:|--------------:| | vroom | 1.5.1 | 1.36 | 53.30 | 1.23 GB/sec | | data.table | 1.14.0 | 5.83 | 12.40 | 281.65 MB/sec | | readr | 1.4.0 | 37.30 | 1.94 | 44.02 MB/sec | | read.delim | 4.1.0 | 72.31 | 1.00 | 22.71 MB/sec | ## Features vroom has nearly all of the parsing features of [readr](https://readr.tidyverse.org) for delimited and fixed width files, including - delimiter guessing\* - custom delimiters (including multi-byte\* and Unicode\* delimiters) - specification of column types (including type guessing) - numeric types (double, integer, big integer\*, number) - logical types - datetime types (datetime, date, time) - categorical types (characters, factors) - column selection, like [`dplyr::select()`](https://dplyr.tidyverse.org/reference/select.html)\* - skipping headers, comments and blank lines - quoted fields - double and backslashed escapes - whitespace trimming - windows newlines - [reading from multiple files or connections\*](#reading-multiple-files) - embedded newlines in headers and fields\*\* - writing delimited files with as-needed quoting. - robust to invalid inputs (vroom has been extensively tested with the [afl](https://lcamtuf.coredump.cx/afl/) fuzz tester)\*. \* *these are additional features not in readr.* \*\* *requires `num_threads = 1`.* ## Installation Install vroom from CRAN with: ``` r install.packages("vroom") ``` Alternatively, if you need the development version from [GitHub](https://github.com/) install it with: ``` r # install.packages("pak") pak::pak("tidyverse/vroom") ``` ## Usage See [getting started](https://vroom.tidyverse.org/articles/vroom.html) to jump start your use of vroom! vroom uses the same interface as readr to specify column types. ``` r vroom::vroom( "mtcars.tsv", col_types = list( cyl = "i", gear = "f", hp = "i", disp = "_", drat = "_", vs = "l", am = "l", carb = "i" ) ) #> # A tibble: 32 Γ— 10 #> model mpg cyl hp wt qsec vs am gear carb #> #> 1 Mazda RX4 21 6 110 2.62 16.5 FALSE TRUE 4 4 #> 2 Mazda RX4 Wag 21 6 110 2.88 17.0 FALSE TRUE 4 4 #> 3 Datsun 710 22.8 4 93 2.32 18.6 TRUE TRUE 4 1 #> # β„Ή 29 more rows ``` ## Reading multiple files vroom natively supports reading from multiple files (or even multiple connections!). First we generate some files to read by splitting the nycflights dataset by airline. For the sake of the example, we’ll just take the first 2 lines of each file. ``` r library(nycflights13) purrr::iwalk( split(flights, flights$carrier), \(x, y) { x$carrier[[1]] vroom::vroom_write( head(x, 2), glue::glue("flights_{y}.tsv"), delim = "\t" ) } ) ``` Then we can efficiently read them into one tibble by passing the filenames directly to vroom. The `id` argument can be used to request a column that reveals the filename that each row originated from. ``` r files <- fs::dir_ls(glob = "flights*tsv") files #> flights_9E.tsv flights_AA.tsv flights_AS.tsv flights_B6.tsv flights_DL.tsv #> flights_EV.tsv flights_F9.tsv flights_FL.tsv flights_HA.tsv flights_MQ.tsv #> flights_OO.tsv flights_UA.tsv flights_US.tsv flights_VX.tsv flights_WN.tsv #> flights_YV.tsv vroom::vroom(files, id = "source") #> Rows: 32 Columns: 20 #> ── Column specification ──────────────────────────────────────────────────────── #> Delimiter: "\t" #> chr (4): carrier, tailnum, origin, dest #> dbl (14): year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, ... #> dttm (1): time_hour #> #> β„Ή Use `spec()` to retrieve the full column specification for this data. #> β„Ή Specify the column types or set `show_col_types = FALSE` to quiet this message. #> # A tibble: 32 Γ— 20 #> source year month day dep_time sched_dep_time dep_delay arr_time #> #> 1 flights_9E.tsv 2013 1 1 810 810 0 1048 #> 2 flights_9E.tsv 2013 1 1 1451 1500 -9 1634 #> 3 flights_AA.tsv 2013 1 1 542 540 2 923 #> # β„Ή 29 more rows #> # β„Ή 12 more variables: sched_arr_time , arr_delay , carrier , #> # flight , tailnum , origin , dest , air_time , #> # distance , hour , minute , time_hour ``` ## Learning more - [Getting started with vroom](https://vroom.tidyverse.org/articles/vroom.html) - [πŸ“½ vroom: Because Life is too short to read slow](https://www.youtube.com/watch?v=RA9AjqZXxMU&t=10s) - Presentation at UseR!2019 ([slides](https://speakerdeck.com/jimhester/vroom)) - [πŸ“Ή vroom: Read and write rectangular data quickly](https://www.youtube.com/watch?v=ZP_y5eaAc60) - a video tour of the vroom features. ## Benchmarks The speed quoted above is from a real 1.53G dataset with 14,388,451 rows and 11 columns, see the [benchmark article](https://vroom.tidyverse.org/articles/benchmarks.html) for full details of the dataset and [bench/](https://github.com/tidyverse/vroom/tree/main/inst/bench) for the code used to retrieve the data and perform the benchmarks. # Environment variables In addition to the arguments to the [`vroom()`](https://vroom.tidyverse.org/reference/vroom.md) function, you can control the behavior of vroom with a few environment variables. Generally these will not need to be set by most users. - `VROOM_TEMP_PATH` - Path to the directory used to store temporary files when reading from a R connection. If unset defaults to the R session’s temporary directory ([`tempdir()`](https://rdrr.io/r/base/tempfile.html)). - `VROOM_THREADS` - The number of processor threads to use when indexing and parsing. If unset defaults to [`parallel::detectCores()`](https://rdrr.io/r/parallel/detectCores.html). - `VROOM_SHOW_PROGRESS` - Whether to show the progress bar when indexing. Regardless of this setting the progress bar is disabled in non-interactive settings, R notebooks, when running tests with testthat and when knitting documents. - `VROOM_CONNECTION_SIZE` - The size (in bytes) of the connection buffer when reading from connections (default is 128 KiB). - `VROOM_WRITE_BUFFER_LINES` - The number of lines to use for each buffer when writing files (default: 1000). There is also a family of variables to control use of the Altrep framework. These variables can take one of these values: `true`, `false`, `TRUE`, `FALSE`, `1`, or `0`. - `VROOM_USE_ALTREP_NUMERICS` - If true, use Altrep for *all* numeric types (default `false`). There are also individual variables for each type. Currently only `VROOM_USE_ALTREP_CHR` defaults to `true`. - `VROOM_USE_ALTREP_CHR` - `VROOM_USE_ALTREP_FCT` - `VROOM_USE_ALTREP_INT` - `VROOM_USE_ALTREP_BIG_INT` - `VROOM_USE_ALTREP_DBL` - `VROOM_USE_ALTREP_NUM` - `VROOM_USE_ALTREP_LGL` - `VROOM_USE_ALTREP_DTTM` - `VROOM_USE_ALTREP_DATE` - `VROOM_USE_ALTREP_TIME` ## Thanks - [Gabe Becker](https://github.com/gmbecker), [Luke Tierney](https://homepage.divms.uiowa.edu/~luke/) and [Tomas Kalibera](https://github.com/kalibera) for conceiving, Implementing and maintaining the [Altrep framework](https://svn.r-project.org/R/branches/ALTREP/ALTREP.html) - [Romain FranΓ§ois](https://github.com/romainfrancois), whose [Altrepisode](https://web.archive.org/web/20200315075838/https://purrple.cat/blog/2018/10/14/altrep-and-cpp/) package and [related blog-posts](https://web.archive.org/web/20200315075838/https://purrple.cat/blog/2018/10/14/altrep-and-cpp/) were a great guide for creating new Altrep objects in C++. - [Matt Dowle](https://github.com/mattdowle) and the rest of the [Rdatatable](https://github.com/Rdatatable) team, `data.table::fread()` is blazing fast and great motivation to see how fast we could go faster! # Package index ## Read rectangular files These functions parse rectangular files (like csv or fixed-width format) into tibbles. They specify the overall structure of the file, and how each line is divided up into fields. - [`vroom()`](https://vroom.tidyverse.org/reference/vroom.md) : Read a delimited file into a tibble - [`vroom_fwf()`](https://vroom.tidyverse.org/reference/vroom_fwf.md) [`fwf_empty()`](https://vroom.tidyverse.org/reference/vroom_fwf.md) [`fwf_widths()`](https://vroom.tidyverse.org/reference/vroom_fwf.md) [`fwf_positions()`](https://vroom.tidyverse.org/reference/vroom_fwf.md) [`fwf_cols()`](https://vroom.tidyverse.org/reference/vroom_fwf.md) : Read a fixed-width file into a tibble - [`problems()`](https://vroom.tidyverse.org/reference/problems.md) : Retrieve parsing problems ## Write rectangular files These functions write data frames to disk, or to convert them to in-memory strings. - [`vroom_write()`](https://vroom.tidyverse.org/reference/vroom_write.md) : Write a data frame to a delimited file - [`vroom_write_lines()`](https://vroom.tidyverse.org/reference/vroom_write_lines.md) : Write lines to a file - [`vroom_format()`](https://vroom.tidyverse.org/reference/vroom_format.md) : Convert a data frame to a delimited string - [`output_column()`](https://vroom.tidyverse.org/reference/output_column.md) : Preprocess column for output ## Column specification The column specification describes how each column is parsed from a character vector in to a more specific data type. vroom does make an educated guess about the type of each column, but you’ll need override those guesses when it gets them wrong. - [`as.col_spec()`](https://vroom.tidyverse.org/reference/as.col_spec.md) : Coerce to a column specification - [`cols()`](https://vroom.tidyverse.org/reference/cols.md) [`cols_only()`](https://vroom.tidyverse.org/reference/cols.md) [`col_logical()`](https://vroom.tidyverse.org/reference/cols.md) [`col_integer()`](https://vroom.tidyverse.org/reference/cols.md) [`col_big_integer()`](https://vroom.tidyverse.org/reference/cols.md) [`col_double()`](https://vroom.tidyverse.org/reference/cols.md) [`col_character()`](https://vroom.tidyverse.org/reference/cols.md) [`col_skip()`](https://vroom.tidyverse.org/reference/cols.md) [`col_number()`](https://vroom.tidyverse.org/reference/cols.md) [`col_guess()`](https://vroom.tidyverse.org/reference/cols.md) [`col_factor()`](https://vroom.tidyverse.org/reference/cols.md) [`col_datetime()`](https://vroom.tidyverse.org/reference/cols.md) [`col_date()`](https://vroom.tidyverse.org/reference/cols.md) [`col_time()`](https://vroom.tidyverse.org/reference/cols.md) : Create column specification - [`cols_condense()`](https://vroom.tidyverse.org/reference/spec.md) [`spec()`](https://vroom.tidyverse.org/reference/spec.md) : Examine the column specifications for a data frame - [`guess_type()`](https://vroom.tidyverse.org/reference/guess_type.md) : Guess the type of a vector ## Locale controls The β€œlocale” controls all options that vary from country-to-country or language-to-language. This includes things like the character used as the decimal mark, the names of days of the week, and the encoding. See `vignette("locales")` for more details. - [`locale()`](https://vroom.tidyverse.org/reference/locale.md) [`default_locale()`](https://vroom.tidyverse.org/reference/locale.md) : Create locales - [`date_names()`](https://vroom.tidyverse.org/reference/date_names.md) [`date_names_lang()`](https://vroom.tidyverse.org/reference/date_names.md) [`date_names_langs()`](https://vroom.tidyverse.org/reference/date_names.md) : Create or retrieve date names ## Data generation vroom provides a number of functions to generate datasets based on a column specification. These are mainly used for development and benchmarking, but can also be useful for reproducing bugs without requiring the original dataset. - [`gen_tbl()`](https://vroom.tidyverse.org/reference/gen_tbl.md) : Generate a random tibble - [`gen_character()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_double()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_number()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_integer()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_factor()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_time()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_date()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_datetime()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_logical()`](https://vroom.tidyverse.org/reference/generators.md) [`gen_name()`](https://vroom.tidyverse.org/reference/generators.md) : Generate individual vectors of the types supported by vroom ## Misc tools These functions are used as helpers for other functions, or to inspect objects. - [`vroom_lines()`](https://vroom.tidyverse.org/reference/vroom_lines.md) : Read lines from a file - [`vroom_altrep()`](https://vroom.tidyverse.org/reference/vroom_altrep.md) : Show which column types are using Altrep - [`vroom_example()`](https://vroom.tidyverse.org/reference/vroom_example.md) [`vroom_examples()`](https://vroom.tidyverse.org/reference/vroom_example.md) : Get path to vroom examples - [`vroom_progress()`](https://vroom.tidyverse.org/reference/vroom_progress.md) : Determine whether progress bars should be shown - [`vroom_str()`](https://vroom.tidyverse.org/reference/vroom_str.md) : Structure of objects # Articles ### All vignettes - [Vroom Benchmarks](https://vroom.tidyverse.org/articles/benchmarks.md): - [Get started with vroom](https://vroom.tidyverse.org/articles/vroom.md):