Before you can wrangle any data you have to get it into R, and getting it out again matters just as much. Three objectives today. First, reading datasets in the four formats you will actually meet: csv, Excel, STATA, and R’s own rds. Second, writing R objects back out to files. And third, which is the one people underestimate, understanding where R looks for a file. That third one is not a section of its own. It sits inside the csv slide, on the tab called where R looks for your files, because a path is an argument to the very functions you are learning there. It is also the part that goes wrong in assignments, because code that works on your machine and nowhere else is not much use to anybody.
Read datasets in various formats (csv, xls, dta, and rds) containing corn yields in Nebraska counties
Write R objects as files in various formats
Understand where R looks for a file, so that your code runs on someone else’s computer too
Two things to do before we go further. Clone the repository linked here, which holds the datasets this lecture uses, so you can follow along rather than watch. And install two packages: tidyverse, which is a whole collection we will lean on for the rest of the course, and haven, which is what reads STATA files. Install them once now. In the code on screen, c combines the two package names into the character vector that install dot packages expects. Eval false is important here because installing is a one-time setup task, so the command is shown without being rerun every time the slides render. The callout makes the point that tidyverse does a great deal more than reading and writing files; today we are using one small corner of it.
tidyverse and haven packages, which we will use to read and write files.Note
The tidyverse package does far more than read and write files. We will cover it in much greater depth later.
How do you know which function to use? You look at the file extension, the bit after the final dot, because that tells you how the data is stored. A csv is plain text that essentially every piece of software can open. An xlsx is Excel’s own format, and note it can hold more than one sheet, which matters later. A dta is STATA’s format, which you will meet constantly if you work around economists. And an rds is R’s own. The rule that follows is simple but absolute: you must use the function that matches the format. There is no universal read-this-file function.
You can tell how a dataset is stored from the file extension, the part that follows the final dot in the file name.
corn.csv: a plain text format that Microsoft Excel and almost everything else supports.
corn.xlsx: Microsoft Excel’s own format, which may hold more than one sheet.
corn.dta: a format that STATA supports (software that is immensely popular among economists).
corn.rds: a format that R supports.
To import a dataset, you must use the function appropriate for the format that dataset is stored in.
Two functions read a csv file, and their names are almost the same, which is a real source of confusion. The first is read.csv, spelled with a dot. It is built into R as part of the utils package, so there is nothing to install and nothing to load. The second is read_csv, spelled with an underscore, and it comes from the readr package, which is part of the tidyverse. Notice that naming convention, because it runs through the whole tidyverse: underscores rather than dots. From the outside the two behave the same way. You hand the function a path to a file and it hands you back your data. The syntax block shows that shape for both. The left arrow stores the returned data under the name on the left, and the quoted text is where the path goes. That block is marked eval false because path to the file to import is not a real file, so those lines are there to be read rather than run. The double colon in readr double-colon read-c-s-v is asking for that function directly from readr, so the line works without a separate library readr call. The example chunk underneath does run. It reads corn-yields-dot-c-s-v twice, once with each function, storing the first result as corn-yields-d-f and the second as corn-yields-t-b-l, and message false suppresses readr’s column-specification message so the slide stays focused on the data itself. What those two objects actually are is not the same, and the Compare tab is about that. But first look at what we typed as the path: a bare file name, with no folders in front of it. That works only when the file is sitting in the one folder R happens to be looking in, and it is the simplest of several ways to write a path. The next tab is about the rest of them.
Two functions read a csv file, and their names are nearly identical:
read.csv(), with a dot, from the utils package that comes with Rread_csv(), with an underscore, from the readr package in the tidyverseSyntax
Example
The quoted text is the path to the file. Here it is a bare file name, which only works when the file sits in the folder R is currently looking in.
A path is the address of a file, and the bare file name on the previous tab is only the simplest kind. These tabs run from the kind of path that works on your own computer only to the kind that works on anybody’s, and end with what to do when R cannot find the file.
Both functions on the previous tab take a path, and a path is simply the address of a file on your computer. There are two kinds, so start with the plainer one. An absolute path begins at the very top of your disk and names every folder on the way down to the file, which means it identifies exactly one file no matter what R is doing or which folder R happens to be pointing at. The first line is what one looks like on a Mac or on Linux: it starts with a slash, then Users, then taro, then Documents, then my-project, then data, and finally corn-yields-dot-c-s-v. The second line is the same idea on Windows, where the path starts with the drive letter C and a colon instead of a bare slash. On both lines the left arrow stores the imported data under the name corn, and eval false keeps them from running when the deck is rendered, because neither of those folders exists on your computer or on mine. That is exactly the problem the callout describes. An absolute path always works, on the one machine it was written on. Send the file to a classmate, or to me, or open it on another computer of your own, and the folders named in it are not there, so the line fails. Nobody else has a folder called taro inside Users. So absolute paths are worth recognising, and worth avoiding in anything you hand in. The remaining tabs are the alternatives.
An absolute path starts at the top of your disk and names every folder down to the file, so it always points at the same file, whatever R is doing.
It works on exactly one computer
Nobody else has a folder called taro inside Users. Send that code to a classmate and every line that reads a file fails. This is the most common reason an assignment that works on your machine does not run on mine.
Now the second kind of path, and the one that starts to make your code portable. A relative path does not begin at the top of the disk. It is read starting from one particular folder, the folder R currently treats as here, which is called the working directory. The function get-w-d tells you what that folder is at the moment. It takes no arguments and prints the full path R is using. The path on the slide is only an illustration of what one looks like, so the one you get back will read differently, and that is fine. Run it yourself, because whatever comes back is where R starts looking for every path that does not begin at the top of the disk. Then read the callout, because the answer depends on where you ask the question. At the Console prompt you get your project root. Inside a chunk in one of your own qmd files you get the folder that qmd file is saved in, both when you click Run and when you render the document. That is the rule you met in the Quarto lecture, and the way to convince yourself is to run get-w-d in both places and compare the two answers. Once you know which folder R is using, the list of relative paths makes sense. A bare file name, corn-yields-dot-c-s-v with no folders in front of it, means the file sits in the working directory itself, and that is what the examples on the previous tab assumed. Data slash the file name goes down into a subfolder called data. Two dots followed by a slash goes up one level, into the folder that contains the working directory. And the two combine: up one level, then down into a folder called data. That third bullet also notes that two dots means the folder above this one, and that you can repeat it, so two dots slash two dots slash goes up two levels. These are worth being fluent in, because a sensible project keeps data and code in separate folders, and relative paths are how they find each other.
A relative path does not start at the top of the disk. It is read from the one folder R currently treats as “here”, called the working directory. getwd() tells you which folder that is:
The answer depends on where you run it
At the Console prompt you get your project root. Inside a chunk in your own .qmd, you get the folder that .qmd is saved in, whether you click Run or render. Run it in both places and compare. See “Console vs. render” in 02-1.
Every path below is measured from that folder:
"corn_yields.csv": the working directory itself, which is what the examples on the previous tab assume"data/corn_yields.csv": the data folder inside the working directory"../corn_yields.csv": one folder up from the working directory. .. means “the folder above this one”, and repeats: "../../corn_yields.csv" goes up two"../data/corn_yields.csv": up one, then into dataAn R Project is the piece that makes all of this reliable, and if you take one practical habit from this lecture, make it this one. Creating a project sets the working directory for you, automatically, every time you open it, so you are never guessing where R thinks it is. You make one from the blue box with a plus in the upper left of RStudio, either starting a new folder or adopting one you already have. Do the exercise in the callout: make a project, close it, reopen it, and run getwd to confirm. This gives your project a fixed root, which is exactly what the here package, two tabs from now, measures every path from.
An R Project sets the working directory for you, every time you open it.
Let’s try
getwd() and confirm it points at the project folderIt gives your project a fixed root, which is what here(), two tabs on, measures every path from.
A tab for the Windows half of the room, because this bites in the first week. Windows displays paths with backslashes, but inside an R string a backslash is a special character that means the next character should be treated differently. So copying a path straight out of File Explorer and pasting it into quotes gives you something that errors, and the error message does not obviously point at backslashes. More precisely, a backslash starts an escape sequence, and in the first example the one before Users is treated as an escape instead of as a folder separator. Eval false is necessary because this line is deliberately showing the broken form. Two fixes, both perfectly acceptable: swap them for forward slashes, which R accepts on Windows, or double every backslash. The two non-running lines show those fixes and both describe the same file. And as the callout says, using here avoids the whole problem, because you never write that part of the path at all.
Windows shows paths with backslashes, but a backslash means something special inside an R string. Copying a path straight from File Explorer gives you this, which errors:
Two fixes, both fine:
Note
here(), in the next tab, sidesteps this entirely, because you never write the top part of the path.
And here is the recommendation, which you first saw in Chapter 2. Inside an R Project, the here package builds paths starting from the project root rather than from wherever R happens to be pointing. You give it the folder and file names as separate pieces and it assembles a correct path for your operating system. The double colon calls here directly from the here package. In the first chunk, the three quoted arguments are successive path pieces: the lectures folder, the Chapter-3-DataWrangling folder, and corn-yields-dot-c-s-v. The printed output is the complete operating-system path that here assembled. In the second chunk, here builds data slash corn-yields-dot-c-s-v and passes that path straight into read-c-s-v, whose result would be stored as corn. Eval false displays the portable pattern without reading the existing project copy. Three reasons this is the one to use: it works no matter which subfolder your qmd sits in, it works on anybody’s computer because nothing above the project root is written down, and it behaves identically whether you run a chunk by hand or render the whole document. If you are ever unsure, run here on its own and it tells you which folder it decided was the root. As the callout notes, it finds that root by looking for the R Project file.
Inside an R Project, the here package builds paths from the project root. This is the one to use.
[1] "/Users/taromieno/Teaching/Data-Science-with-R-Quarto/lectures/Chapter-3-DataWrangling/corn_yields.csv"Why:
Note
Run here::here() on its own to see which folder it decided is the root. It looks for your .Rproj file.
This tab is a checklist for the single most common error in this chapter, the one that says your file does not exist in the current working directory. Work through it in order rather than guessing. First, getwd: where is R actually looking, as opposed to where you assume it is. Second, list.files: what does R actually see in that folder, compared against the name you typed. Third, check the extension, and this catches a lot of people, because Windows hides extensions by default, so what looks like a csv may really have a dot txt on the end. Fourth, check spelling and capitalization, since on some systems capital and lowercase are different files. The specific error on screen names corn-yields-dot-c-s-v. Both get-w-d and list-dot-files take no arguments here; the first prints the full folder path and the second prints the names inside it, so compare those outputs character for character against what you typed. Eval false means the calls are displayed for you to run when you need them, without adding machine-specific folder listings to the rendered slide.
The error looks like this:
Error: 'corn_yields.csv' does not exist in current working directoryWork through these in order:
getwd(): where is R actually looking?list.files(): what does R see in that folder? Compare against the name you typed.corn_yields.csv may really be corn_yields.csv.txt.Corn_Yields.csv and corn_yields.csv are different files on some systems.Back to those two functions. So what actually differs between them, and does it matter to you? Three things. First, what you get back: read.csv gives a data.frame, read_csv gives a tibble. The two class calls produce those labels. A tibble’s class output can contain several inherited class names, including t-b-l-d-f, t-b-l, and data-dot-frame, and this readr result may also begin with spec-t-b-l-d-f. Those names describe one tibble with layers of behavior, not several separate objects. Second, how it prints: a tibble shows ten rows and tells you the type of every column, whereas a data.frame dumps the whole thing into your console, which is miserable for a large dataset. Third, and this is the one that bites, awkward column names. If your file has a column called county space name, read.csv silently renames it with a dot, while read_csv leaves it alone. Because spaces cannot be used as ordinary bare names in R code, you refer to that preserved name by surrounding it with backticks. The callout gives the recommendation: use read_csv. It is faster, it does not rename things behind your back, and a tibble is what the rest of this course expects.
1. What you get back
2. How it prints
A tibble shows the first 10 rows and the type of every column. A data.frame prints the entire object, which floods the screen for a large dataset.
3. What happens to awkward column names
If the file has a column called county name, with a space:
read.csv() renames it to county.nameread_csv() keeps it as county name, and you refer to it with backticksWhich should you use?
Use read_csv(). It is faster on large files, it does not rename your columns behind your back, and it tells you which type it guessed for each column. This course uses the tidyverse throughout, so tibble is the object you want anyway.
Now a real-world problem the clean examples hide. read_csv guesses each column’s type by sampling up to guess_max rows spread throughout the file, and sometimes it guesses wrong. The classic case is a zip code or a county code that looks like a number, so it is read as one, and the leading zero disappears. Once that has happened you cannot get it back. The fix is to stop it guessing and tell it directly, using col_types. In the code, cols defines the column specification, county-code names the column we are overriding, and col-character tells readr to preserve its values as text, including any leading zero. Columns you do not name there are still guessed. Eval false makes this a demonstration rather than trying to reread the lecture file with a column it does not actually contain. Three other arguments worth remembering: skip, for files with a title sitting above the header row, na, for telling it which strings mean missing, and col_names FALSE when there is no header at all. Here, skip equals two ignores exactly the first two lines; na uses c to collect the empty string, uppercase N-A, and a dot as three different missing-value markers; and col-names FALSE says the first row is data rather than names.
read_csv() guesses each column’s type from the first rows. When it guesses wrong, say a zip code read as a number and stripped of its leading zero, tell it directly:
Other arguments worth knowing:
skip = 2: ignore the first 2 lines, for files with a title above the headerna = c("", "NA", "."): treat these strings as missingcol_names = FALSE: the file has no header rowA small convenience with a warning attached. Anywhere read_csv accepts a file path, it also accepts a web address, so you can read a dataset straight off the internet without downloading it first. In the example, read-c-s-v receives the complete raw-file U-R-L as its one argument, and the left arrow would store the downloaded data as corn. Eval false keeps this illustrative address from being contacted when the deck renders. That is genuinely useful when you are exploring. But read the callout before you build anything on it: this re-downloads the file every single time you render, which is slow, and more importantly a URL can change or vanish without warning, taking your analysis with it. For anything you actually rely on, download it once, keep it in your project, and read the local copy.
read_csv() accepts a web address wherever it accepts a file path:
Important
This re-downloads the file every time you render. For anything you rely on, download it once, save it in your project, and read the local copy. A URL can change or disappear without warning.
Now Excel files. The function is read_excel from the readxl package, and there is one wrinkle worth stating clearly because it catches everybody. readxl is installed when you install tidyverse, so you already have it. But it is not loaded when you call library of tidyverse, unlike most of the others. So you have to load it explicitly with its own library call. If you get a could not find function read_excel error despite having tidyverse loaded, this is why.
You can use read_excel() from the readxl package to read data sheets from an xls(x) file.
readxl is installed when you install the tidyverse package, but it is not loaded when you run library(tidyverse).
So you have to load it yourself:
The syntax adds one argument you have not needed before: sheet. An Excel workbook can hold many sheets, so you have to say which one you want, and the simplest way is by number, with sheet equals one for the first. The examples show reading two different sheets from the same workbook into two different objects. That is the basic pattern, and it works. But referring to sheets by number has a weakness, which is what the next tab is about.
Syntax
Examples
Here is that weakness. Sheet numbers are positional, so the moment somebody inserts a sheet at the front of the workbook, every number shifts by one and your code silently reads the wrong data. Silently is the problem; nothing errors. So use names instead. The function excel_sheets lists every sheet name in a workbook without you having to open Excel at all, which is useful in its own right. Its file argument is corn-yields-dot-x-l-s, and the output here shows corn-yields-zero-eight and corn-yields-zero-nine. Then pass that name to the sheet argument instead of a number. The second chunk passes corn-yields-zero-eight, reads that named sheet from the same workbook, and stores it as corn-zero-eight. It reads better and it survives somebody reorganizing the file.
Referring to a sheet by number is fragile: insert a sheet and every number shifts. Use the name instead.
First, list what is in the workbook:
Then read the one you want:
One short observation to close the Excel section. Check the class of what came back: it is a tibble, exactly the same kind of object read_csv produced. The class call asks R what kind of object corn-zero-eight is, and its output lists t-b-l-d-f, t-b-l, and data-dot-frame. The first two labels identify a tibble, while data-dot-frame appears because a tibble is built on top of a data.frame. That is worth noticing because it means everything you learn about working with tibbles applies regardless of what file format the data started life in. The format matters at the moment of reading and then stops mattering, which is a genuinely good property of the tidyverse tools.
The data comes in as a tibble, the same object read_csv() produces.
STATA files, which have a dta extension, are read with read_dta from the haven package. If you work anywhere near economics you will meet these constantly, because STATA remains the standard tool in a lot of that world. Load haven, call read_dta with a path, and you have your data. The library call attaches haven so its functions are available by name. The syntax template is marked eval false because its quoted path is only a placeholder. The double colon in haven double-colon read-d-t-a is the other way to reach the function: it asks for read-d-t-a directly from haven, even without relying on the earlier library call. In the running example, corn-yields-dot-d-t-a is the input path and the left arrow stores the imported object as corn-yields. There is nothing more to the basic case than that. The last chunk checks what you were handed, and the class call prints three things: tbl_df, tbl, and data.frame. So what did you actually get? A tibble, the same as everywhere else. The reason data.frame appears at the end of that list is explained in the callout, and it is worth understanding properly because it causes real confusion. A tibble is built on top of a data.frame, with extra behaviour layered over it, so it truthfully reports that it is also a data.frame. Seeing data.frame in that list does not mean you have a plain data.frame.
Use the read_dta() function from the haven package.
Syntax
Example
read_dta() returns a tibble, just like read_csv() and read_excel().
Why data.frame still shows up in the class
A tibble reports three classes: tbl_df, tbl, and data.frame. The last one is there because a tibble is a data.frame underneath, with extra behavior layered on top. Seeing data.frame in the list does not mean you have a plain data.frame.
The last format is R’s own, the rds file, and it is different in kind from the others. An rds file stores a single R object exactly as it is, and only R can read it. Note the wording carefully: a single R object, not necessarily a dataset. That distinction matters, and we come back to it when we export. Reading one needs no packages at all, because readRDS is built into R. This is the format you will use for saving your own intermediate results. The syntax is the simplest of the lot: readRDS, and a path. The first chunk is marked eval false because path to the file is only a placeholder. In the running example, read-R-D-S opens corn-yields-dot-r-d-s and the left arrow stores the returned object as corn-yields. No sheet to specify, no package to load, no column types to worry about, because the file already contains a complete R object, so there is nothing to interpret or guess at. It simply hands the object back to you. The last chunk checks the class, and you get a tibble again, but for a completely different reason than before. The output lists t-b-l-d-f, t-b-l, and data-dot-frame, which identifies corn-yields as that tibble. With csv, the function decided to build you a tibble. Here, the object that was originally saved into this file happened to be a tibble, and an rds file gives you back precisely what was put into it. No conversion, no guessing, no types lost along the way. That fidelity is the whole selling point of the format, and it is why it is the right choice for saving work in progress.
An rds file stores a single R object exactly as it is, and is read only by R.
You can use the readRDS() function to read an rds file. No special packages are necessary.
Syntax
Example
The imported dataset is already a tibble, because the object saved into corn_yields.rds was a tibble, and an rds file gives you back exactly what was put in.
Exporting is the mirror image of importing, and pleasantly the naming follows the same pattern, so you already half know it. Write-c-s-v writes a data.frame or tibble as csv, write-d-t-a writes one in STATA’s dta format, and save-R-D-S saves an R object as rds. That last one also has a readr alternative, write-r-d-s, which serves the same basic purpose with tidyverse-style naming and its own defaults. Pick whichever reads better to you. The general shape is the same across all of them, which is the next tab.
Exporting datasets works much the same way as importing them.
Here is the list of functions that let you export a data.frame or tibble in different formats:
write_csv()write_dta()saveRDS() (or write_rds() from readr, which is the same thing with tidyverse-style argument order)The syntax for every export function is the same: the object you want to save, then the file name to save it as. Object first, destination second. The examples show all three formats writing the same data. In each example, corn-yields is the object. Readr double-colon write-c-s-v creates corn-yields-exp-dot-c-s-v, haven double-colon write-d-t-a creates corn-yields-exp-dot-d-t-a, and save-R-D-S creates corn-yields-exp-dot-r-d-s. The double colons select functions from packages without attaching those packages. Eval false matters especially on an export slide because it shows the commands without creating or overwriting those three files whenever the deck renders. The comments separate the examples by format and are not commands. Now read the callout about Excel, because it is a genuine recommendation rather than a preference. A spreadsheet is a thing for people to look at, not a thing for code to read; saving results into one puts them somewhere that is harder to verify and trivially easy for somebody to edit by hand without leaving a trace. If a supervisor genuinely requires one, write_xlsx exists. The callout’s writexl double-colon write-x-l-s-x form reaches that function directly from the writexl package.
Syntax
Examples
Writing Excel files
Avoid writing Excel files. A spreadsheet is for people to look at, not for code to read, and saving results into one puts them in a format that is harder to check and easy to edit by hand. If you genuinely need one, use writexl::write_xlsx().
Two things worth knowing about rds specifically, in the tabs here. The first is that it preserves any R object, not just datasets. The example saves a list containing both a character vector and a whole dataset, and gets the identical list back. Try to do that with a csv and you cannot even begin. The second tab answers the practical question of which format to choose. rds is smallest, but size is the least interesting consideration. The real trade-off is that csv can be opened by anything in twenty years but forgets every type, while rds preserves everything perfectly but only R can read it. The callout gives a sensible default: keep raw data as csv, save intermediate results as rds.
Look first at Object type preservation. The list call creates a-list with two named elements: a is the character vector R and rocks, made with c, and b is the entire corn-yields object. The next line asks temp-dir for R’s temporary folder, then file-dot-path safely joins that folder to a-list-dot-r-d-s and stores the resulting path as r-d-s-path. That path exists so the demonstration does not leave an output file in the course folder. Save-R-D-S writes the whole list to that path, and read-R-D-S reads it back and prints it. The output shows that a is still a character vector and b is still the dataset with its column types. A csv cannot do that because it can only represent a rectangular table of text values.
Now move to Which format should you use. The data-dot-frame call builds the comparison table on screen. The file column is a character vector of the three file names. File-dot-size returns each size in bytes, division by one thousand twenty-four converts those values to kibibytes, and round with one as its second argument keeps one decimal place before the values become the K-i-B column. For these files, the output is fourteen-point-seven kibibytes for csv, eight-point-eight for dta, and one-point-five for rds, so rds is the smallest. The size comparison is not the only decision. Csv stays broadly readable but loses stored types, rds preserves the exact R object but only R can read it, and dta is the choice when the recipient works in STATA. That is why the callout recommends csv for raw data and rds for intermediate results whose types need to survive between scripts.
You can export any kind of R object as an rds file, not just a dataset.
a_list <- list(a = c("R", "rocks"), b = corn_yields)
rds_path <- file.path(tempdir(), "a_list.rds")
saveRDS(a_list, rds_path)
readRDS(rds_path)$a
[1] "R" "rocks"
$b
# A tibble: 161 × 9
Year State FIPS County_name State_name Commodity `Data item` Irrigated
<int> <int> <int> <chr> <chr> <chr> <chr> <int>
1 2008 31 31019 BUFFALO NEBRASKA CORN CORN, GRAIN - Y… 0
2 2008 31 31019 BUFFALO NEBRASKA CORN CORN, GRAIN, IR… 1
3 2008 31 31041 CUSTER NEBRASKA CORN CORN, GRAIN - Y… 0
4 2008 31 31041 CUSTER NEBRASKA CORN CORN, GRAIN, IR… 1
5 2008 31 31047 DAWSON NEBRASKA CORN CORN, GRAIN - Y… 0
6 2008 31 31047 DAWSON NEBRASKA CORN CORN, GRAIN, IR… 1
7 2008 31 31077 GREELEY NEBRASKA CORN CORN, GRAIN - Y… 0
8 2008 31 31077 GREELEY NEBRASKA CORN CORN, GRAIN, IR… 1
9 2008 31 31079 HALL NEBRASKA CORN CORN, GRAIN - Y… 0
10 2008 31 31079 HALL NEBRASKA CORN CORN, GRAIN, IR… 1
# ℹ 151 more rows
# ℹ 1 more variable: Yield <int>A list goes in, and a list comes back out, with every column type intact.
file KiB
1 corn_yields.csv 14.7
2 corn_yields.dta 8.8
3 corn_yields.rds 1.5rds is the smallest, because it is compressed and stores R’s own representation.
But size is the least important consideration:
A reasonable default
Keep your raw data as csv, so it stays readable by anything. Save intermediate results as rds, so the types survive between scripts.