Before you can wrangle any data you have to get it into R, and getting it out again matters just as much. Three objectives today. First, reading datasets in the four formats you will actually meet: csv, Excel, STATA, and R’s own rds. Second, writing R objects back out to files. And third, which is the one people underestimate, understanding where R looks for a file. That third objective is the middle third of this lecture and it is the part that goes wrong in assignments, because code that works on your machine and nowhere else is not much use to anybody.
Read datasets in various formats (csv, xls, dta, and rds) containing corn yields in Nebraska counties
Write R objects as files in various formats
Understand where R looks for a file, so that your code runs on someone else’s computer too
Two things to do before we go further. Clone the repository linked here, which holds the datasets this lecture uses, so you can follow along rather than watch. And install two packages: tidyverse, which is a whole collection we will lean on for the rest of the course, and haven, which is what reads STATA files. Install them once now. In the code on screen, c combines the two package names into the character vector that install dot packages expects. Eval false is important here because installing is a one-time setup task, so the command is shown without being rerun every time the slides render. The callout makes the point that tidyverse does a great deal more than reading and writing files; today we are using one small corner of it.
tidyverse and haven packages, which we will use to read and write files.Note
The tidyverse package does far more than read and write files. We will cover it in much greater depth later.
How do you know which function to use? You look at the file extension, the bit after the final dot, because that tells you how the data is stored. A csv is plain text that essentially every piece of software can open. An xlsx is Excel’s own format, and note it can hold more than one sheet, which matters later. A dta is STATA’s format, which you will meet constantly if you work around economists. And an rds is R’s own. The rule that follows is simple but absolute: you must use the function that matches the format. There is no universal read-this-file function.
You can tell how a dataset is stored from the file extension, the part that follows the final dot in the file name.
corn.csv: a plain text format that Microsoft Excel and almost everything else supports.
corn.xlsx: Microsoft Excel’s own format, which may hold more than one sheet.
corn.dta: a format that STATA supports (software that is immensely popular among economists).
corn.rds: a format that R supports.
To import a dataset, you must use the function appropriate for the format that dataset is stored in.
Start with csv, and with the function that comes built into R: read.csv, with a dot. You give it a path to a file and it hands you back a data.frame. The syntax block shows the shape, and the example actually runs, reading the corn yields file. In that syntax block, the quoted text is where the file path goes, and the left arrow stores the returned data under the name on the left. Eval false keeps that placeholder example from running because path to the file to import is not a real file. The second chunk reads corn-yields-dot-c-s-v and assigns the result to corn-yields-d-f. Notice that this example uses only the file name. That works when the file is in R’s working directory, and we will soon look closely at how R decides what that directory is. This function works, it needs no extra packages, and for a small clean file it is perfectly adequate. But it is not what I am going to recommend.
You can use read.csv() from the utils package, which comes with R.
Syntax
Example
Here is the alternative: read_csv, with an underscore, from the readr package, which comes with tidyverse. Notice the naming convention, because it runs through the whole tidyverse: underscores rather than dots. The call looks almost identical, you hand it a path and get your data back. The double colon in readr double-colon read-c-s-v asks for that function directly from readr, so this line does not depend on a separate library readr call. The first chunk is a non-running syntax template. In the example, the file name is the input, the left arrow saves the imported data as corn-yields-t-b-l, and message false suppresses readr’s column-specification message so the slide stays focused on the object itself. The difference is in what you get back and how it behaves, and that is what the next tab is about. When you see both of these in code on the internet, and you will, the underscore version is the modern one.
You can use read_csv() from the readr package.
Syntax
Example
So what actually differs, and does it matter to you? Three things. First, what you get back: read.csv gives a data.frame, read_csv gives a tibble. The two class calls produce those labels. A tibble’s class output can contain several inherited class names, including t-b-l-d-f, t-b-l, and data-dot-frame, and this readr result may also begin with spec-t-b-l-d-f. Those names describe one tibble with layers of behavior, not several separate objects. Second, how it prints: a tibble shows ten rows and tells you the type of every column, whereas a data.frame dumps the whole thing into your console, which is miserable for a large dataset. Third, and this is the one that bites, awkward column names. If your file has a column called county space name, read.csv silently renames it with a dot, while read_csv leaves it alone. Because spaces cannot be used as ordinary bare names in R code, you refer to that preserved name by surrounding it with backticks. The callout gives the recommendation: use read_csv. It is faster, it does not rename things behind your back, and a tibble is what the rest of this course expects.
1. What you get back
2. How it prints
A tibble shows the first 10 rows and the type of every column. A data.frame prints the entire object, which floods the screen for a large dataset.
3. What happens to awkward column names
If the file has a column called county name, with a space:
read.csv() renames it to county.nameread_csv() keeps it as county name, and you refer to it with backticksWhich should you use?
Use read_csv(). It is faster on large files, it does not rename your columns behind your back, and it tells you which type it guessed for each column. This course uses the tidyverse throughout, so tibble is the object you want anyway.
Now a real-world problem the clean examples hide. read_csv guesses each column’s type by sampling up to guess_max rows spread throughout the file, and sometimes it guesses wrong. The classic case is a zip code or a county code that looks like a number, so it is read as one, and the leading zero disappears. Once that has happened you cannot get it back. The fix is to stop it guessing and tell it directly, using col_types. In the code, cols defines the column specification, county-code names the column we are overriding, and col-character tells readr to preserve its values as text, including any leading zero. Columns you do not name there are still guessed. Eval false makes this a demonstration rather than trying to reread the lecture file with a column it does not actually contain. Three other arguments worth remembering: skip, for files with a title sitting above the header row, na, for telling it which strings mean missing, and col_names FALSE when there is no header at all. Here, skip equals two ignores exactly the first two lines; na uses c to collect the empty string, uppercase N-A, and a dot as three different missing-value markers; and col-names FALSE says the first row is data rather than names.
read_csv() guesses each column’s type from the first rows. When it guesses wrong, say a zip code read as a number and stripped of its leading zero, tell it directly:
Other arguments worth knowing:
skip = 2: ignore the first 2 lines, for files with a title above the headerna = c("", "NA", "."): treat these strings as missingcol_names = FALSE: the file has no header rowA small convenience with a warning attached. Anywhere read_csv accepts a file path, it also accepts a web address, so you can read a dataset straight off the internet without downloading it first. In the example, read-c-s-v receives the complete raw-file U-R-L as its one argument, and the left arrow would store the downloaded data as corn. Eval false keeps this illustrative address from being contacted when the deck renders. That is genuinely useful when you are exploring. But read the callout before you build anything on it: this re-downloads the file every single time you render, which is slow, and more importantly a URL can change or vanish without warning, taking your analysis with it. For anything you actually rely on, download it once, keep it in your project, and read the local copy.
read_csv() accepts a web address wherever it accepts a file path:
Important
This re-downloads the file every time you render. For anything you rely on, download it once, save it in your project, and read the local copy. A URL can change or disappear without warning.
Now the section that matters most for making your code work somewhere other than your own laptop. R always has one folder it treats as here, called the working directory, and any path that does not start from the root of your disk is interpreted relative to it. getwd tells you what it currently is; run it and look. It takes no arguments and prints the full path R is currently using, so compare that output with the folder you expect. Once you know that, the simple case makes sense: if the working directory is your project folder and the file is sitting in it, then the bare file name is enough. The example’s bare name is corn-yields-dot-c-s-v, the left arrow would store the imported data as corn-yield, and eval false leaves that line as an example rather than importing the file again on this tab. Everything else in this section is about controlling that folder deliberately rather than by accident.
R always has one folder it treats as “here”, called the working directory. A path that does not start from the root of your disk is interpreted relative to it.
So if the working directory is your project folder, this is enough:
You do not have to keep every file in one folder. A relative path lets you point at nearby folders starting from the working directory. Data slash the filename goes into a subfolder called data. Two dots followed by a slash goes up one level, to the folder containing your working directory. And you can combine them, going up one and then down into a different folder. The callout notes you can repeat the two dots to go up several levels. These are worth being fluent in, because a sensible project has data and code in separate folders, and relative paths are how they find each other.
You can point at files in nearby folders without writing the whole path:
"data/corn_yields.csv": the data folder inside the working directory"../corn_yields.csv": one folder up from the working directory"../data/corn_yields.csv": up one, then into dataNote
.. means “the folder above this one”. You can repeat it: "../../corn_yields.csv" goes up two levels.
An R Project is the piece that makes all of this reliable, and if you take one practical habit from this lecture, make it this one. Creating a project sets the working directory for you, automatically, every time you open it, so you are never guessing where R thinks it is. You make one from the blue box with a plus in the upper left of RStudio, either starting a new folder or adopting one you already have. Do the exercise in the callout: make a project, close it, reopen it, and run getwd to confirm. This gives your project a fixed root, which is exactly what the next tab builds on.
An R Project sets the working directory for you, every time you open it.
Let’s try
getwd() and confirm it points at the project folderThis is the foundation for the next tab: it gives your project a fixed root that everything else can be measured from.
And here is the recommendation, which you first saw in Chapter 2. Inside an R Project, the here package builds paths starting from the project root rather than from wherever R happens to be pointing. You give it the folder and file names as separate pieces and it assembles a correct path for your operating system. The double colon calls here directly from the here package. In the first chunk, the three quoted arguments are successive path pieces: the lectures folder, the Chapter-3-DataWrangling folder, and corn-yields-dot-c-s-v. The printed output is the complete operating-system path that here assembled. In the second chunk, here builds data slash corn-yields-dot-c-s-v and passes that path straight into read-c-s-v, whose result would be stored as corn. Eval false displays the portable pattern without reading the existing project copy. Three reasons this is the one to use: it works no matter which subfolder your qmd sits in, it works on anybody’s computer because nothing above the project root is written down, and it behaves identically whether you run a chunk by hand or render the whole document. If you are ever unsure, run here on its own and it tells you which folder it decided was the root. As the callout notes, it finds that root by looking for the R Project file.
Inside an R Project, the here package builds paths from the project root:
[1] "/Users/taromieno/Teaching/Data-Science-with-R-Quarto/lectures/Chapter-3-DataWrangling/corn_yields.csv"Why this is the one to use:
Note
Run here::here() on its own to see which folder it decided is the root. It looks for your .Rproj file.
A tab for the Windows half of the room, because this bites in the first week. Windows displays paths with backslashes, but inside an R string a backslash is a special character that means the next character should be treated differently. So copying a path straight out of File Explorer and pasting it into quotes gives you something that errors, and the error message does not obviously point at backslashes. More precisely, a backslash starts an escape sequence, and in the first example the one before Users is treated as an escape instead of as a folder separator. Eval false is necessary because this line is deliberately showing the broken form. Two fixes, both perfectly acceptable: swap them for forward slashes, which R accepts on Windows, or double every backslash. The two non-running lines show those fixes and both describe the same file. And as the callout says, using here avoids the whole problem, because you never write that part of the path at all.
Windows shows paths with backslashes, but a backslash means something special inside an R string. Copying a path straight from File Explorer gives you this, which errors:
Two fixes, both fine:
Note
Using here() sidesteps this entirely, because you never write the top part of the path.
This tab is a checklist for the single most common error in this chapter, the one that says your file does not exist in the current working directory. Work through it in order rather than guessing. First, getwd: where is R actually looking, as opposed to where you assume it is. Second, list.files: what does R actually see in that folder, compared against the name you typed. Third, check the extension, and this catches a lot of people, because Windows hides extensions by default, so what looks like a csv may really have a dot txt on the end. Fourth, check spelling and capitalization, since on some systems capital and lowercase are different files. The specific error on screen names corn-yields-dot-c-s-v. Both get-w-d and list-dot-files take no arguments here; the first prints the full folder path and the second prints the names inside it, so compare those outputs character for character against what you typed. Eval false means the calls are displayed for you to run when you need them, without adding machine-specific folder listings to the rendered slide.
The error looks like this:
Error: 'corn_yields.csv' does not exist in current working directoryWork through these in order:
getwd(): where is R actually looking?list.files(): what does R see in that folder? Compare against the name you typed.corn_yields.csv may really be corn_yields.csv.txt.Corn_Yields.csv and corn_yields.csv are different files on some systems.Now Excel files. The function is read_excel from the readxl package, and there is one wrinkle worth stating clearly because it catches everybody. readxl is installed when you install tidyverse, so you already have it. But it is not loaded when you call library of tidyverse, unlike most of the others. So you have to load it explicitly with its own library call. If you get a could not find function read_excel error despite having tidyverse loaded, this is why.
You can use read_excel() from the readxl package to read data sheets from an xls(x) file.
readxl is installed when you install the tidyverse package, but it is not loaded when you run library(tidyverse).
So you have to load it yourself:
The syntax adds one argument you have not needed before: sheet. An Excel workbook can hold many sheets, so you have to say which one you want, and the simplest way is by number, with sheet equals one for the first. The examples show reading two different sheets from the same workbook into two different objects. That is the basic pattern, and it works. But referring to sheets by number has a weakness, which is what the next tab is about.
Syntax
Examples
Here is that weakness. Sheet numbers are positional, so the moment somebody inserts a sheet at the front of the workbook, every number shifts by one and your code silently reads the wrong data. Silently is the problem; nothing errors. So use names instead. The function excel_sheets lists every sheet name in a workbook without you having to open Excel at all, which is useful in its own right. Its file argument is corn-yields-dot-x-l-s, and the output here shows corn-yields-zero-eight and corn-yields-zero-nine. Then pass that name to the sheet argument instead of a number. The second chunk passes corn-yields-zero-eight, reads that named sheet from the same workbook, and stores it as corn-zero-eight. It reads better and it survives somebody reorganizing the file.
Referring to a sheet by number is fragile: insert a sheet and every number shifts. Use the name instead.
First, list what is in the workbook:
Then read the one you want:
One short observation to close the Excel section. Check the class of what came back: it is a tibble, exactly the same kind of object read_csv produced. The class call asks R what kind of object corn-zero-eight is, and its output lists t-b-l-d-f, t-b-l, and data-dot-frame. The first two labels identify a tibble, while data-dot-frame appears because a tibble is built on top of a data.frame. That is worth noticing because it means everything you learn about working with tibbles applies regardless of what file format the data started life in. The format matters at the moment of reading and then stops mattering, which is a genuinely good property of the tidyverse tools.
The data comes in as a tibble, the same object read_csv() produces.
STATA files, which have a dta extension, are read with read_dta from the haven package. If you work anywhere near economics you will meet these constantly, because STATA remains the standard tool in a lot of that world. Load haven, call read_dta with a path, and you have your data. The library call attaches haven so its functions are available by name. The syntax template is marked eval false because its quoted path is only a placeholder. The double colon in haven double-colon read-d-t-a is the other way to reach the function: it asks for read-d-t-a directly from haven, even without relying on the earlier library call. In the running example, corn-yields-dot-d-t-a is the input path and the left arrow stores the imported object as corn-yields. There is nothing more to the basic case than that. The two tabs after this one cover the two things about STATA files that do occasionally surprise people.
Use the read_dta() function from the haven package.
Syntax
Example
Check the class and you will see three things listed: tbl_df, tbl, and data.frame. So what did you actually get? A tibble, the same as everywhere else. The reason data.frame appears at the end of that list is explained in the callout, and it is worth understanding properly because it causes real confusion. A tibble is built on top of a data.frame, with extra behaviour layered over it, so it truthfully reports that it is also a data.frame. Seeing data.frame in that list does not mean you have a plain data.frame.
read_dta() returns a tibble, just like read_csv() and read_excel().
Why data.frame still shows up in the class
A tibble reports three classes: tbl_df, tbl, and data.frame. The last one is there because a tibble is a data.frame underneath, with extra behavior layered on top. Seeing data.frame in the list does not mean you have a plain data.frame.
One genuine oddity of STATA files. STATA lets you attach labels to variables, and haven preserves them, which means a column can come back as a special labelled type rather than a plain number. Most of the time this is invisible and harmless. But occasionally a column will refuse to behave in arithmetic or will plot strangely, and the labels are why. The fix is one function, zap_labels, which strips them off and leaves you with ordinary columns. The code calls it directly from haven, passes in the whole corn-yields object, and assigns the cleaned result back to that same name. Eval false keeps this as a remedy to use when you actually encounter labelled columns, rather than stripping information from the lecture object automatically. Worth filing away, because when it happens the symptoms are baffling if you have never heard of it.
STATA files often carry labels that R keeps as a haven_labelled column. If a column will not cooperate in arithmetic or plotting, strip the labels:
The last format is R’s own, the rds file, and it is different in kind from the others. An rds file stores a single R object exactly as it is, and only R can read it. Note the wording carefully: a single R object, not necessarily a dataset. That distinction matters, and we come back to it when we export. Reading one needs no packages at all, because readRDS is built into R. This is the format you will use for saving your own intermediate results.
An rds file stores a single R object exactly as it is, and is read only by R.
You can use the readRDS() function to read an rds file.
No special packages are necessary.
The syntax is the simplest of the lot: readRDS, and a path. The first chunk is marked eval false because path to the file is only a placeholder. In the running example, read-R-D-S opens corn-yields-dot-r-d-s and the left arrow stores the returned object as corn-yields. No sheet to specify, no package to load, no column types to worry about. That is because the file already contains a complete R object, so there is nothing to interpret or guess at. It simply hands the object back to you. Which is exactly the point of the format, and it leads directly into the note on the next tab.
Syntax
Example
Check the class here and you get a tibble again, but for a completely different reason than before. The output lists t-b-l-d-f, t-b-l, and data-dot-frame, which identifies corn-yields as that tibble. With csv, the function decided to build you a tibble. Here, the object that was originally saved into this file happened to be a tibble, and an rds file gives you back precisely what was put into it. No conversion, no guessing, no types lost along the way. That fidelity is the whole selling point of the format, and it is why it is the right choice for saving work in progress.
The imported dataset is already a tibble. That is because the object that was saved into corn_yields.rds was a tibble, and an rds file gives you back exactly what was put in.
Exporting is the mirror image of importing, and pleasantly the naming follows the same pattern, so you already half know it. Write-c-s-v writes a data.frame or tibble as csv, write-d-t-a writes one in STATA’s dta format, and save-R-D-S saves an R object as rds. That last one also has a readr alternative, write-r-d-s, which serves the same basic purpose with tidyverse-style naming and its own defaults. Pick whichever reads better to you. The general shape is the same across all of them, which is the next tab.
Exporting datasets works much the same way as importing them.
Here is the list of functions that let you export a data.frame or tibble in different formats:
write_csv()write_dta()saveRDS() (or write_rds() from readr, which is the same thing with tidyverse-style argument order)The syntax for every export function is the same: the object you want to save, then the file name to save it as. Object first, destination second. The examples show all three formats writing the same data. In each example, corn-yields is the object. Readr double-colon write-c-s-v creates corn-yields-exp-dot-c-s-v, haven double-colon write-d-t-a creates corn-yields-exp-dot-d-t-a, and save-R-D-S creates corn-yields-exp-dot-r-d-s. The double colons select functions from packages without attaching those packages. Eval false matters especially on an export slide because it shows the commands without creating or overwriting those three files whenever the deck renders. The comments separate the examples by format and are not commands. Now read the callout about Excel, because it is a genuine recommendation rather than a preference. A spreadsheet is a thing for people to look at, not a thing for code to read; saving results into one puts them somewhere that is harder to verify and trivially easy for somebody to edit by hand without leaving a trace. If a supervisor genuinely requires one, write_xlsx exists. The callout’s writexl double-colon write-x-l-s-x form reaches that function directly from the writexl package.
Syntax
Examples
Writing Excel files
Avoid writing Excel files. A spreadsheet is for people to look at, not for code to read, and saving results into one puts them in a format that is harder to check and easy to edit by hand. If you genuinely need one, use writexl::write_xlsx().
Two things worth knowing about rds specifically, in the tabs here. The first is that it preserves any R object, not just datasets. The example saves a list containing both a character vector and a whole dataset, and gets the identical list back. Try to do that with a csv and you cannot even begin. The second tab answers the practical question of which format to choose. rds is smallest, but size is the least interesting consideration. The real trade-off is that csv can be opened by anything in twenty years but forgets every type, while rds preserves everything perfectly but only R can read it. The callout gives a sensible default: keep raw data as csv, save intermediate results as rds.
Look first at Object type preservation. The list call creates a-list with two named elements: a is the character vector R and rocks, made with c, and b is the entire corn-yields object. The next line asks temp-dir for R’s temporary folder, then file-dot-path safely joins that folder to a-list-dot-r-d-s and stores the resulting path as r-d-s-path. That path exists so the demonstration does not leave an output file in the course folder. Save-R-D-S writes the whole list to that path, and read-R-D-S reads it back and prints it. The output shows that a is still a character vector and b is still the dataset with its column types. A csv cannot do that because it can only represent a rectangular table of text values.
Now move to Which format should you use. The data-dot-frame call builds the comparison table on screen. The file column is a character vector of the three file names. File-dot-size returns each size in bytes, division by one thousand twenty-four converts those values to kibibytes, and round with one as its second argument keeps one decimal place before the values become the K-i-B column. For these files, the output is fourteen-point-seven kibibytes for csv, eight-point-eight for dta, and one-point-five for rds, so rds is the smallest. The size comparison is not the only decision. Csv stays broadly readable but loses stored types, rds preserves the exact R object but only R can read it, and dta is the choice when the recipient works in STATA. That is why the callout recommends csv for raw data and rds for intermediate results whose types need to survive between scripts.
You can export any kind of R object as an rds file, not just a dataset.
a_list <- list(a = c("R", "rocks"), b = corn_yields)
rds_path <- file.path(tempdir(), "a_list.rds")
saveRDS(a_list, rds_path)
readRDS(rds_path)$a
[1] "R" "rocks"
$b
# A tibble: 161 × 9
Year State FIPS County_name State_name Commodity `Data item` Irrigated
<int> <int> <int> <chr> <chr> <chr> <chr> <int>
1 2008 31 31019 BUFFALO NEBRASKA CORN CORN, GRAIN - Y… 0
2 2008 31 31019 BUFFALO NEBRASKA CORN CORN, GRAIN, IR… 1
3 2008 31 31041 CUSTER NEBRASKA CORN CORN, GRAIN - Y… 0
4 2008 31 31041 CUSTER NEBRASKA CORN CORN, GRAIN, IR… 1
5 2008 31 31047 DAWSON NEBRASKA CORN CORN, GRAIN - Y… 0
6 2008 31 31047 DAWSON NEBRASKA CORN CORN, GRAIN, IR… 1
7 2008 31 31077 GREELEY NEBRASKA CORN CORN, GRAIN - Y… 0
8 2008 31 31077 GREELEY NEBRASKA CORN CORN, GRAIN, IR… 1
9 2008 31 31079 HALL NEBRASKA CORN CORN, GRAIN - Y… 0
10 2008 31 31079 HALL NEBRASKA CORN CORN, GRAIN, IR… 1
# ℹ 151 more rows
# ℹ 1 more variable: Yield <int>A list goes in, and a list comes back out, with every column type intact.
data.frame(
file = c("corn_yields.csv", "corn_yields.dta", "corn_yields.rds"),
KiB = round(file.size(
c("corn_yields.csv", "corn_yields.dta", "corn_yields.rds")
) / 1024, 1)
) file KiB
1 corn_yields.csv 14.7
2 corn_yields.dta 8.8
3 corn_yields.rds 1.5rds is the smallest, because it is compressed and stores R’s own representation.
But size is the least important consideration:
A reasonable default
Keep your raw data as csv, so it stays readable by anything. Save intermediate results as rds, so the types survive between scripts.