Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scraper Update #41

Merged
merged 2 commits into from
May 16, 2024
Merged

Scraper Update #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
995b55e
Update README example.
milistu May 16, 2024
d25ea6e
Add coments to run_scraper.
milistu May 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion scraper/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,12 @@ python scraper/scraper.py --file scraper/urls.txt --output-dir laws_test
- `--output-dir`: Directory to save the JSON files (default is scraper/laws).

## Example
To scrape law articles from a list of URLs in urls.txt and save the output in the `scraper/laws` directory:
To scrape law articles from a single URL (example: Serbian Labor Law) and save the output in the `scraper/laws` directory:
```bash
python scraper/scraper.py --url "https://www.paragraf.rs/propisi/zakon_o_radu.html" --output-dir scraper/laws
```

To scrape law articles from a list of URLs in urls.txt and save the output in the `scraper/laws` directory:
```bash
python scraper/scraper.py --file scraper/urls.txt --output-dir scraper/laws
```
Expand Down
8 changes: 8 additions & 0 deletions scraper/scraper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,18 @@ def run_scraper(soup: BeautifulSoup, url: str) -> List[Dict]:
article_texts = []
article_link = None

# Find all <p> elements in the HTML
elements = soup.find_all("p")
for el in elements:
# Determine the class name of the element
class_name = (
"clan" if check_class_element(element=el, class_name="clan") else "normal"
)

# If the element is a title (class "clan"), start a new article
if class_name == "clan":
if article_title:
# Save the previous article
law_articles.append(
{
"title": article_title,
Expand All @@ -54,13 +58,17 @@ def run_scraper(soup: BeautifulSoup, url: str) -> List[Dict]:
}
)
article_texts = []
# Get the article title
article_title = el.get_text(strip=True)

# Get the link to the article section
name_attr = el.find("a").get("name") if el.find("a") else None
article_link = f"{url}#{name_attr}" if name_attr else None
# If the element is part of an article's text, add it to the current article
elif article_title and class_name == "normal":
article_texts.append(el.get_text(strip=True))

# Save the last article
if article_title and article_texts:
law_articles.append(
{"title": article_title, "texts": article_texts, "link": article_link}
Expand Down