You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The package provides a simple ‘robotstxt’ class and accompanying methods
17
+
to parse and check ‘robots.txt’ files. Data fields are provided as data
18
+
frames and vectors. Permissions can be checked by providing path
19
+
character vectors and optional bot names.
16
20
17
-
The package provides a simple 'robotstxt' class and accompanying methods to parse and check 'robots.txt' files. Data fields are provided as data frames and vectors. Permissions can be checked by providing path character vectors and optional bot names.
18
-
19
-
20
21
# Robots.txt files
21
22
22
-
Robots.txt files are a way to kindly ask webbots, spiders, crawlers, wanderers and the like to access or not access certain parts of a webpage. The de facto 'standard' never made it beyond a informal ["Network Working Group INTERNET DRAFT"](http://www.robotstxt.org/norobots-rfc.txt). Nonetheless, the use of robots.txt files is widespread (e.g. https://en.wikipedia.org/robots.txt, https://www.google.com/robots.txt) and bots from Google, Yahoo and the like will adhere to the rules defined in robots.txt files - although, their interpretation of those rules might differ (e.g. [rules for googlebot ](https://developers.google.com/search/reference/robots_txt)).
23
-
24
-
As the name of the files already suggests robots.txt files are plain text and always found at the root of a domain. The syntax of the files in essence follows a `fieldname: value` scheme with optional preceding `user-agent: ...` lines to indicate the scope of the following rule block. Blocks are separated by blank lines and the omission of a user-agent field (which directly corresponds to the HTTP user-agent field) is seen as referring to all bots. `#` serves to comment lines and parts of lines. Everything after `#` until the end of line is regarded a comment. Possible field names are: user-agent, disallow, allow, crawl-delay, sitemap, and host.
25
-
26
-
27
-
Let us have an example file to get an idea how a robots.txt file might look like. The file below starts with a comment line followed by a line disallowing access to any content -- everything that is contained in root ("`/`") -- for all bots. The next block concerns GoodBot and NiceBot. Those two get the previous permissions lifted by being disallowed nothing. The third block is for PrettyBot. PrettyBot likes shiny stuff and therefor gets a special permission for everything contained in the "`/shinystuff/`" folder while all other restrictions still hold. In the last block all bots are asked to pause at least 5 seconds between two visits.
28
-
29
-
30
-
```robots.txt
31
-
# this is a comment
32
-
# a made up example of an robots.txt file
33
-
34
-
Disallow: /
35
-
36
-
User-agent: GoodBot # another comment
37
-
User-agent: NiceBot
38
-
Disallow:
39
-
40
-
User-agent: PrettyBot
41
-
Allow: /shinystuff/
42
-
43
-
Crawl-Delay: 5
44
-
```
45
-
46
-
For more information have a look at: http://www.robotstxt.org/norobots-rfc.txt, where the robots.txt file 'standard' is described formally. Valuable introductions can be found at http://www.robotstxt.org/robotstxt.html as well as at https://en.wikipedia.org/wiki/Robots_exclusion_standard - of cause.
23
+
Robots.txt files are a way to kindly ask webbots, spiders, crawlers,
24
+
wanderers and the like to access or not access certain parts of a
25
+
webpage. The de facto ‘standard’ never made it beyond a informal
26
+
[“Network Working Group INTERNET
27
+
DRAFT”](http://www.robotstxt.org/norobots-rfc.txt). Nonetheless, the use
28
+
of robots.txt files is widespread
29
+
(e.g. <https://en.wikipedia.org/robots.txt>,
30
+
<https://www.google.com/robots.txt>) and bots from Google, Yahoo and the
31
+
like will adhere to the rules defined in robots.txt files - although,
32
+
their interpretation of those rules might differ (e.g. [rules for
As the name of the files already suggests robots.txt files are plain
36
+
text and always found at the root of a domain. The syntax of the files
37
+
in essence follows a `fieldname: value` scheme with optional preceding
38
+
`user-agent: ...` lines to indicate the scope of the following rule
39
+
block. Blocks are separated by blank lines and the omission of a
40
+
user-agent field (which directly corresponds to the HTTP user-agent
41
+
field) is seen as referring to all bots. `#` serves to comment lines and
42
+
parts of lines. Everything after `#` until the end of line is regarded a
43
+
comment. Possible field names are: user-agent, disallow, allow,
44
+
crawl-delay, sitemap, and host.
45
+
46
+
Let us have an example file to get an idea how a robots.txt file might
47
+
look like. The file below starts with a comment line followed by a line
48
+
disallowing access to any content – everything that is contained in root
49
+
(“`/`”) – for all bots. The next block concerns GoodBot and NiceBot.
50
+
Those two get the previous permissions lifted by being disallowed
51
+
nothing. The third block is for PrettyBot. PrettyBot likes shiny stuff
52
+
and therefor gets a special permission for everything contained in the
53
+
“`/shinystuff/`” folder while all other restrictions still hold. In the
54
+
last block all bots are asked to pause at least 5 seconds between two
55
+
visits.
56
+
57
+
# this is a comment
58
+
# a made up example of an robots.txt file
59
+
60
+
Disallow: /
61
+
62
+
User-agent: GoodBot # another comment
63
+
User-agent: NiceBot
64
+
Disallow:
65
+
66
+
User-agent: PrettyBot
67
+
Allow: /shinystuff/
68
+
69
+
Crawl-Delay: 5
70
+
71
+
For more information have a look at:
72
+
<http://www.robotstxt.org/norobots-rfc.txt>, where the robots.txt file
73
+
‘standard’ is described formally. Valuable introductions can be found at
74
+
<http://www.robotstxt.org/robotstxt.html> as well as at
75
+
<https://en.wikipedia.org/wiki/Robots_exclusion_standard> - of cause.
47
76
48
77
# Fast food usage for the uninterested
49
78
50
-
```{r, message=FALSE}
51
-
library(robotstxt)
52
-
paths_allowed("http://google.com/")
53
-
paths_allowed("http://google.com/search")
54
-
```
79
+
library(robotstxt)
80
+
paths_allowed("http://google.com/")
55
81
82
+
## [1] TRUE
56
83
84
+
paths_allowed("http://google.com/search")
57
85
58
-
# Example Usage
86
+
## [1] FALSE
59
87
60
-
First, let us load the package. In addition we load the dplyr package to be able to use the magrittr pipe operator `%>%` and some easy to read and remember data manipulation functions.
88
+
# Example Usage
61
89
62
-
```{r, message=FALSE}
63
-
library(robotstxt)
64
-
library(dplyr)
65
-
```
66
-
67
-
## object oriented style
90
+
First, let us load the package. In addition we load the dplyr package to
91
+
be able to use the magrittr pipe operator `%>%` and some easy to read
92
+
and remember data manipulation functions.
68
93
69
-
The first step is to create an instance of the robotstxt class provided by the package. The instance has to be initiated via providing either domain or the actual text of the robots.txt file. If only the domain is provided, the robots.txt file will be downloaded automatically. Have a look at `?robotstxt` for descriptions of all data fields and methods as well as their parameters.
94
+
library(robotstxt)
95
+
library(dplyr)
70
96
97
+
## object oriented style
71
98
72
-
```{r, include=FALSE}
73
-
rtxt <-
74
-
robotstxt(
75
-
domain = "wikipedia.org",
76
-
text = robotstxt:::rt_get_rtxt("robots_wikipedia.txt")
77
-
)
78
-
```
99
+
The first step is to create an instance of the robotstxt class provided
100
+
by the package. The instance has to be initiated via providing either
101
+
domain or the actual text of the robots.txt file. If only the domain is
102
+
provided, the robots.txt file will be downloaded automatically. Have a
103
+
look at `?robotstxt` for descriptions of all data fields and methods as
104
+
well as their parameters.
79
105
80
-
```{r, eval=FALSE}
81
-
rtxt <- robotstxt(domain="wikipedia.org")
82
-
```
106
+
rtxt <- robotstxt(domain="wikipedia.org")
83
107
84
108
`rtxt` is of class `robotstxt`.
85
109
86
-
```{r}
87
-
class(rtxt)
88
-
```
89
-
90
-
Printing the object lets us glance at all data fields and methods in `rtxt` - we have access to the text as well as all common fields. Non-standard fields are collected in `other`.
91
-
92
-
```{r}
93
-
rtxt
94
-
```
95
-
96
-
Checking permissions works via `rtxt`'s `check` method by providing one or more paths. If no bot name is provided `"*"` - meaning any bot - is assumed.
Printing the object lets us glance at all data fields and methods in
115
+
`rtxt` - we have access to the text as well as all common fields.
116
+
Non-standard fields are collected in `other`.
117
+
118
+
rtxt
119
+
120
+
## $text
121
+
## [1] "#\n# robots.txt for http://www.wikipedia.org/ and friends\n#\n# Please note: There are a lot of pages on this site, and there are\n# some misbehaved spiders out there that go _way_ too fast. If you're\n# irresponsible, your access to the site may be blocked.\n#\n\n# advertising-related bots:\nUser-agent: Mediapartners-Google*\n\n[... 653 lines omitted ...]"
While working with the robotstxt class is recommended the checking can be done with functions only as well. In the following we (1) download the robots.txt file; (2) parse it and (3) check permissions.
0 commit comments