python_notes/파이썬 15-2.BeautifulSoup.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"toc_visible":true,"authorship_tag":"ABX9TyN2AReFXy4w3FSe7a1Q/zuH"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["#22036의 Python 시험대비 교재\n","* 이 교재는 여러 수준의 독자를 위해 제작되었습니다\n","* 오류나 지적사항, 건의사항은 카톡이나 [여기로](https://forms.gle/cLfQGtNjQmg2hAjm7) 문의해주세요\n","* 이해가 가지 않는 부분이나 추가 설명(추가 예시라던가 예제라던가)이 필요하시면 마찬가지로 문의해주세요\n","* '~~알고리즘 정리해주세요' '풀이방법 정리해주세요' 같은 요구사항도 마찬가지로\n","* 피드백 적극 반영합니다\n","* ~~문의좀 해줘요 자습하기싫어~~"],"metadata":{"id":"SLoCnBoATnMH"}},{"cell_type":"markdown","source":["# BeautifulSoup 사용하기\n","\n","BeautifulSoup은 웹크롤링 및 HTML 또는 XML 문서 구문 분석에 사용되는 Python 라이브러리입니다. HTML 구조를 탐색하고 조작하여 웹 페이지에서 데이터를 추출하는 방법을 제공합니다. BeautifulSoup의 다양한 특징과 기능을 자세히 살펴보겠습니다."],"metadata":{"id":"20hneuIg9mvc"}},{"cell_type":"markdown","source":["## BeautifulSoup 설치\n","\n","BeautifulSoup을 사용하려면 먼저 설치해야 합니다. Python용 패키지 설치 프로그램인 pip를 사용하여 BeautifulSoup을 설치할 수 있습니다. 명령 프롬프트 또는 터미널을(Windows 기준 작업 표시줄 검색창에 cmd/명령 프롬프트 or Win+R → cmd) 열고 다음 명령을 실행합니다.\n","\n","```\n","pip install beautifulsoup4\n","```\n","\n","설치가 완료되면 다음 코드를 사용하여 라이브러리를 Python 스크립트로 가져올 수 있습니다.\n","\n","```python\n","from bs4 import BeautifulSoup\n","```"],"metadata":{"id":"0uwDucjmCcVI"}},{"cell_type":"markdown","source":["## HTML 파싱하기\n","\n","웹페이지의 구조는 HTML로 이루어져 있습니다. HTML 문서에서 정보 추출을 시작하려면 먼저 BeautifulSoup을 사용하여 HTML 콘텐츠를 구문 분석(파싱)해야 합니다. BeautifulSoup은 lxml, html5lib 및 내장 HTML 파서와 같은 다양한 파서를 지원합니다.\n","\n","다음은 HTML 문서를 파싱하는 예시입니다:\n"],"metadata":{"id":"TD1AzEbTDouv"}},{"cell_type":"code","source":["from bs4 import BeautifulSoup\n","\n","# 파싱할 HTML\n","html_doc = \"\"\"\n","<html>\n","<head>\n","    <title>Example Website</title>\n","</head>\n","<body>\n","    <h1>Welcome to BeautifulSoup</h1>\n","    <p>This is a sample paragraph.</p>\n","</body>\n","</html>\n","\"\"\"\n","\n","# BeautifulSoup 객체 만들기\n","soup = BeautifulSoup(html_doc, 'html.parser')\n","\n","# HTML 구조 접근해서 갖고놀기\n","print(soup.title.text)  # Example Website\n","print(soup.h1.text)  # Welcome to BeautifulSoup\n","print(soup.p.text)  # This is a sample paragraph."],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"qKC2NSNUKN-V","executionInfo":{"status":"ok","timestamp":1686459926811,"user_tz":-540,"elapsed":411,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"502ebdcb-dd3c-4f07-8eec-a040e51b649a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Example Website\n","Welcome to BeautifulSoup\n","This is a sample paragraph.\n"]}]},{"cell_type":"markdown","source":["위 예제에서는 HTML 내용과 사용할 구문 분석기를 전달하여 `soup`라는 BeautifulSoup 개체를 만듭니다. `soup` 개체를 사용하면 HTML 구조에 접근하고 조작할 수 있습니다."],"metadata":{"id":"Qse-z_2nOQJn"}},{"cell_type":"markdown","source":["## HTML 구조 탐색\n","\n","BeautifulSoup를 이용해 HTML 구조를 탐색하고 특정 요소에 액세스하는 할 수 있습니다. 이러한 메서드를 사용하면 이름, 속성 또는 CSS 클래스를 기반으로 특정 HTML 태그를 검색할 수 있습니다.\n","일반적으로 사용되는 몇 가지 방법은 다음과 같습니다.\n","* **`find()`**: 지정된 기준과 일치하는 태그의 첫 번째 항목을 찾을 수 있습니다.\n","* **`find_all()`**: 지정된 기준과 일치하는 모든 태그의 리스트를 반환합니다.\n","* **`parent`, `find_parent()`, `parents`**: 지정된 태그의 부모(상위) 요소에 접근할 수 있습니다.\n","* **`next_sibling`, `previous_sibling`, `find_next_sibling()`, `find_previous_sibling()`**: 지정된 태그의 다음 또는 이전 형제 요소에 접근할 수 있습니다.\n","\n","다음은 BeautifulSoup을 사용하여 HTML 구조를 탐색하는 방법을 보여주는 예입니다."],"metadata":{"id":"4umSi7eoHv0P"}},{"cell_type":"code","source":["from bs4 import BeautifulSoup\n","\n","# 아래 HTML 구조를 가정합니다.\n","html_doc = \"\"\"\n","<div class=\"container\">\n","    <h2>Example Heading</h2>\n","    <ul>\n","        <li class=\"item\">Item 1</li>\n","        <li class=\"item\">Item 2</li>\n","        <li class=\"item\">Item 3</li>\n","    </ul>\n","</div>\n","\"\"\"\n","soup = BeautifulSoup(html_doc, \"html.parser\")\n","\n","# \"container\"를 클래스로 하는 div요소 찾기\n","container = soup.find(\"div\", class_=\"container\")\n","print(\"container:\")\n","print(container)\n","print()\n","\n","# div 내에서 h2 요소 찾기\n","heading = container.find(\"h2\")\n","print(\"heading:\")\n","print(heading)\n","print()\n","\n","# h2 요소의 텍스트에 접근\n","print(\"heading.text:\")\n","print(heading.text)\n","print()\n","\n","# ul 내의 모든 li 요소 찾기\n","items = container.ul.find_all(\"li\")\n","print(\"items:\")\n","print(items)\n","print()\n","\n","# 반복해서 출력\n","for item in items:\n","    print(item.text)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"PT3Sq3qho4K5","executionInfo":{"status":"ok","timestamp":1686459944164,"user_tz":-540,"elapsed":3,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"6e66fe6d-1d23-4b66-aee2-5ad0aaaebff2"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["container:\n","<div class=\"container\">\n","<h2>Example Heading</h2>\n","<ul>\n","<li class=\"item\">Item 1</li>\n","<li class=\"item\">Item 2</li>\n","<li class=\"item\">Item 3</li>\n","</ul>\n","</div>\n","\n","heading:\n","<h2>Example Heading</h2>\n","\n","heading.text:\n","Example Heading\n","\n","items:\n","[<li class=\"item\">Item 1</li>, <li class=\"item\">Item 2</li>, <li class=\"item\">Item 3</li>]\n","\n","Item 1\n","Item 2\n","Item 3\n"]}]},{"cell_type":"markdown","source":["위 예시에서는 `find()` 메서드를 사용하여 `\"container\"` 클래스가 있는 `div` 요소를 찾습니다. 거기에서 우리는 `div` 내의 `h2` 요소에 더 탐색하고 액세스할 수 있을 뿐만 아니라 `ul` 내의 모든 `li` 요소를 검색할 수 있습니다."],"metadata":{"id":"8y3fn2N8qllW"}},{"cell_type":"markdown","source":["위의 `find()`말고도 `select()`메서드를 사용할 수 있습니다. `select()`는 CSS 선택자를 사용하여 요소를 선택할 수 있는 메서드입니다. CSS 선택자는 속성, 클래스 이름 또는 ID를 기반으로 웹 페이지의 특정 요소를 대상으로 지정하는 널리 사용되는 방법입니다. `select()` 메서드는 지정된 CSS 선택자와 일치하는 요소 목록을 반환합니다."],"metadata":{"id":"mvCAe0TIyB_v"}},{"cell_type":"code","source":["from bs4 import BeautifulSoup\n","\n","# 아래 HTML 구조를 가정합니다.\n","html_doc = \"\"\"\n","<div class=\"container\">\n","    <h2>Example Heading</h2>\n","    <ul>\n","        <li class=\"item\">Item 1</li>\n","        <li class=\"item\">Item 2</li>\n","        <li class=\"item\">Item 3</li>\n","    </ul>\n","</div>\n","\"\"\"\n","soup = BeautifulSoup(html_doc, \"html.parser\")\n","\n","# class가 \"item\"인 모든 li요소를 선택합니다.\n","items = soup.select('li.item')\n","for item in items:\n","    print(item.text)  # Item 1, Item 2, Item 3\n","\n","# class가 \"container\"인 div 내의 h2 요소를 선택합니다.\n","heading = soup.select_one('div.container > h2')\n","print(heading.text)  # Output: Example Heading"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Gd6hJ5AE1zZg","executionInfo":{"status":"ok","timestamp":1686459895225,"user_tz":-540,"elapsed":2,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"3b79e6d7-4440-4559-b9e3-de0dd1aac829"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Item 1\n","Item 2\n","Item 3\n","Example Heading\n"]}]},{"cell_type":"markdown","source":["위 코드의 첫 예시에서는 CSS 선택자 `'li.item'`과 함께 `select()` 메서드를 사용하여 `\"item\"` 클래스가 있는 모든 `li` 요소를 선택합니다.\n","\n","두 번째 예시에서는 CSS 선택자 `'div.container > h2'`와 함께 `select_one()` 메서드를 사용하여 `\"container\"` 클래스가 있는 `div`의 직계 자식인 `h2` 요소를 선택합니다. `select_one()` 메서드는 선택자와 일치하는 첫 번째 요소만 반환합니다."],"metadata":{"id":"dVs8x9xZ241Y"}},{"cell_type":"markdown","source":["## BeautifulSoup로 웹 크롤링하기\n","\n","BeautifulSoup 라이브러의 HTML분석을 이용해 온라인상의 웹페이지에서 정보를 수집하는 웹크롤링을 수행할 수 있습니다."],"metadata":{"id":"NGMOOKriVHhy"}},{"cell_type":"markdown","source":["### requests 라이브러리\n","\n","requests 라이브러리는 HTTP 요청을 만드는 데 사용되는 널리 사용되는 Python 라이브러리입니다. HTTP 요청을 보내고 응답을 처리하는 프로세스를 단순화하여 웹 서비스 및 API와 더 쉽게 상호작용할 수 있습니다.\n","\n","다음 코드를 통해 특정 url의 서버에 get 요청을 보낼 수 있습니다."],"metadata":{"id":"EboXTecIemGC"}},{"cell_type":"code","source":["import requests\n","\n","response = requests.get(\"https://www.google.com/\")"],"metadata":{"id":"Te5nlnVQIDkG"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["아래와 같은 방식으로 HTTP 응답 코드를 받아 올 수 있습니다."],"metadata":{"id":"EE8HiVN6IN71"}},{"cell_type":"code","source":["print(response.status_code)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"w02SkCXEIMCW","executionInfo":{"status":"ok","timestamp":1686481492885,"user_tz":-540,"elapsed":495,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"db71f214-d8a5-422f-cf82-430d87aa8ed1"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["200\n"]}]},{"cell_type":"markdown","source":["다음은 일반적으로 많이 볼 수 있는 HTTP 응답 코드입니다.\n","\n","* 200 OK(성공): 성공적으로 처리했을 때. 가장 일반적으로 볼 수 있는 HTTP 상태.\n","* 400 Bad Request(잘못된 요청): 요청 자체가 잘못되었을 때.\n","* 403 Forbidden(거부됨): 서버가 요청을 거부할 때 발생.\n","* 404 Not Found(찾을 수 없음): 찾는 리소스가 없음.\n","* 500 Internal Server Error(내부 서버 오류): 서버에 오류가 발생함.\n"],"metadata":{"id":"lVjkZTXOIWdi"}},{"cell_type":"markdown","source":["`request.get()`으로 서버와 통신하는 데 성공했다면, `text`를 통해 웹의 HTML 구조를 가져올 수 있습니다."],"metadata":{"id":"Lgl1GRXdJM0M"}},{"cell_type":"code","source":["print(response.text)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"MDw6cSOOJYtu","executionInfo":{"status":"ok","timestamp":1686481773339,"user_tz":-540,"elapsed":518,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"6daf9fd4-5750-42ee-cf65-87879ea58814"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"nl\"><head><meta content=\"text/html; charset=UTF-8\" http-equiv=\"Content-Type\"><meta content=\"/images/branding/googleg/1x/googleg_standard_color_128dp.png\" itemprop=\"image\"><title>Google</title><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){var _g={kEI:'K6qFZJThJu7Y7_UPmOKG6A4',kEXPI:'0,18167,1341242,6059,206,4804,896,1420,383,246,5,1129120,1704,1196079,303179,77529,16114,28684,22431,1361,12316,17583,4998,17075,38444,2872,2891,3926,214,4209,3405,606,29877,20182,26966,20583,4,1528,2304,42126,18096,22595,6642,7596,1,42160,2,39755,5679,1020,31123,4568,6258,23416,1254,5835,14968,4332,5016,2468,25076,2006,8155,7381,15969,874,19633,7,1922,9779,12414,10494,19551,2007,1135,17057,927,4869,8,14428,7651,12555,8377,18988,5375,3030,5629,482,9705,1804,10472,2276,609,161,288,10742,8549,10958,44,2983,1428,87,117,7950,4468,1275,168,1128,780,7495,355,666,4,2413,2882,2467,1902,979,732,2824,3,626,6161,2093,3,3,9,4113,643,319,29,280,2041,363,9,1198,557,21,1979,5206187,664,607,49,84,2,4,2,8798145,3311,141,795,19735,1,1,346,6044,98,200,26,4,11,258,12,17,1,1,1,5,5,1,1,1,3,3,7,3,11,3,27,3,19,1,3,23943715,578,2772317,1271212,1008,9920,5745,2893,6250,19356,532,5170,1405738,260490,23499462,85,93,134,2049,4,279,530,292,871,1301,1,548,616,944,1972,1,64,2333,107,228,166,335,1532,14,1724,277,53,1045,16,112,160,80,1110,13,565,13,110,434,5,1012,6,1307,149,26,198,443,48,515,336,11,825,496,213,491,7,477,248,2,171,134,548,614,57,890,817,38,916,151,90,192,359,327,141,2,55,54,1,268,15,216,135,262,38,89,468,345,38,644,123,94,497,440,5,19,1,124,147,69,2,92,476,582,1563,10,639,155,170,522,391,673,3,343,634,122,3,1085',kBL:'I1bB',kOPI:89978449};if (window.google && window.google.stvsc){window.google.kEI = _g.kEI;}else {window.google = _g;}\n","})();(function(){google.sn='webhp';google.kHL='nl';})();(function(){\n","var h=this||self;function l(){return void 0!==window.google&&void 0!==window.google.kOPI&&0!==window.google.kOPI?window.google.kOPI:null};var m,n=[];function p(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||m}function q(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute(\"leid\")));)a=a.parentNode;return b}function r(a){/^http:/i.test(a)&&\"https:\"===window.location.protocol&&(google.ml&&google.ml(Error(\"a\"),!1,{src:a,glmm:1}),a=\"\");return a}\n","function t(a,b,c,d,k){var e=\"\";-1===b.search(\"&ei=\")&&(e=\"&ei=\"+p(d),-1===b.search(\"&lei=\")&&(d=q(d))&&(e+=\"&lei=\"+d));d=\"\";var g=-1===b.search(\"&cshid=\")&&\"slh\"!==a,f=[];f.push([\"zx\",Date.now().toString()]);h._cshid&&g&&f.push([\"cshid\",h._cshid]);c=c();null!=c&&f.push([\"opi\",c.toString()]);for(c=0;c<f.length;c++){if(0===c||0<c)d+=\"&\";d+=f[c][0]+\"=\"+f[c][1]}return\"/\"+(k||\"gen_204\")+\"?atyp=i&ct=\"+String(a)+\"&cad=\"+(b+e+d)};m=google.kEI;google.getEI=p;google.getLEI=q;google.ml=function(){return null};google.log=function(a,b,c,d,k,e){e=void 0===e?l:e;c||(c=t(a,b,e,d,k));if(c=r(c)){a=new Image;var g=n.length;n[g]=a;a.onerror=a.onload=a.onabort=function(){delete n[g]};a.src=c}};google.logUrl=function(a,b){b=void 0===b?l:b;return t(\"\",a,b)};}).call(this);(function(){google.y={};google.sy=[];google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.sx=function(a){google.sy.push(a)};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};google.bx=!1;google.lx=function(){};}).call(this);google.f={};(function(){\n","document.documentElement.addEventListener(\"submit\",function(b){var a;if(a=b.target){var c=a.getAttribute(\"data-submitfalse\");a=\"1\"===c||\"q\"===c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);document.documentElement.addEventListener(\"click\",function(b){var a;a:{for(a=b.target;a&&a!==document.documentElement;a=a.parentElement)if(\"A\"===a.tagName){a=\"1\"===a.getAttribute(\"data-nohref\");break a}a=!1}a&&b.preventDefault()},!0);}).call(this);</script><style>#gbar,#guser{font-size:13px;padding-top:1px !important;}#gbar{height:22px}#guser{padding-bottom:7px !important;text-align:right}.gbh,.gbd{border-top:1px solid #c9d7f1;font-size:1px}.gbh{height:0;position:absolute;top:24px;width:100%}@media all{.gb1{height:22px;margin-right:.5em;vertical-align:top}#gbar{float:left}}a.gb1,a.gb4{text-decoration:underline !important}a.gb1,a.gb4{color:#00c !important}.gbi .gb4{color:#dd8e27 !important}.gbf .gb4{color:#900 !important}\n","</style><style>body,td,a,p,.h{font-family:arial,sans-serif}body{margin:0;overflow-y:scroll}#gog{padding:3px 8px 0}td{line-height:.8em}.gac_m td{line-height:17px}form{margin-bottom:20px}.h{color:#1967d2}em{font-weight:bold;font-style:normal}.lst{height:25px;width:496px}.gsfi,.lst{font:18px arial,sans-serif}.gsfs{font:17px arial,sans-serif}.ds{display:inline-box;display:inline-block;margin:3px 0 4px;margin-left:4px}input{font-family:inherit}body{background:#fff;color:#000}a{color:#681da8;text-decoration:none}a:hover,a:active{text-decoration:underline}.fl a{color:#1967d2}a:visited{color:#681da8}.sblc{padding-top:5px}.sblc a{display:block;margin:2px 0;margin-left:13px;font-size:11px}.lsbb{background:#f8f9fa;border:solid 1px;border-color:#dadce0 #70757a #70757a #dadce0;height:30px}.lsbb{display:block}#WqQANb a{display:inline-block;margin:0 12px}.lsb{background:url(/images/nav_logo229.png) 0 -261px repeat-x;color:#000;border:none;cursor:pointer;height:30px;margin:0;outline:0;font:15px arial,sans-serif;vertical-align:top}.lsb:active{background:#dadce0}.lst:focus{outline:none}</style><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){window.google.erd={jsr:1,bv:1808,de:true};\n","var h=this||self;var k,l=null!=(k=h.mei)?k:1,n,p=null!=(n=h.sdo)?n:!0,q=0,r,t=google.erd,v=t.jsr;google.ml=function(a,b,d,m,e){e=void 0===e?2:e;b&&(r=a&&a.message);if(google.dl)return google.dl(a,e,d),null;if(0>v){window.console&&console.error(a,d);if(-2===v)throw a;b=!1}else b=!a||!a.message||\"Error loading script\"===a.message||q>=l&&!m?!1:!0;if(!b)return null;q++;d=d||{};b=encodeURIComponent;var c=\"/gen_204?atyp=i&ei=\"+b(google.kEI);google.kEXPI&&(c+=\"&jexpid=\"+b(google.kEXPI));c+=\"&srcpg=\"+b(google.sn)+\"&jsr=\"+b(t.jsr)+\"&bver=\"+b(t.bv);var f=a.lineNumber;void 0!==f&&(c+=\"&line=\"+f);var g=\n","a.fileName;g&&(0<g.indexOf(\"-extension:/\")&&(e=3),c+=\"&script=\"+b(g),f&&g===window.location.href&&(f=document.documentElement.outerHTML.split(\"\\n\")[f],c+=\"&cad=\"+b(f?f.substring(0,300):\"No script found.\")));c+=\"&jsel=\"+e;for(var u in d)c+=\"&\",c+=b(u),c+=\"=\",c+=b(d[u]);c=c+\"&emsg=\"+b(a.name+\": \"+a.message);c=c+\"&jsst=\"+b(a.stack||\"N/A\");12288<=c.length&&(c=c.substr(0,12288));a=c;m||google.log(0,\"\",a);return a};window.onerror=function(a,b,d,m,e){r!==a&&(a=e instanceof Error?e:Error(a),void 0===d||\"lineNumber\"in a||(a.lineNumber=d),void 0===b||\"fileName\"in a||(a.fileName=b),google.ml(a,!1,void 0,!1,\"SyntaxError\"===a.name||\"SyntaxError\"===a.message.substring(0,11)||-1!==a.message.indexOf(\"Script error\")?3:0));r=null;p&&q>=l&&(window.onerror=null)};})();</script></head><body bgcolor=\"#fff\"><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){var src='/images/nav_logo229.png';var iesg=false;document.body.onload = function(){window.n && window.n();if (document.images){new Image().src=src;}\n","if (!iesg){document.f&&document.f.q.focus();document.gbqf&&document.gbqf.q.focus();}\n","}\n","})();</script><div id=\"mngb\"><div id=gbar><nobr><b class=gb1>Zoeken</b> <a class=gb1 href=\"https://www.google.nl/imghp?hl=nl&tab=wi\">Afbeeldingen</a> <a class=gb1 href=\"https://maps.google.nl/maps?hl=nl&tab=wl\">Maps</a> <a class=gb1 href=\"https://play.google.com/?hl=nl&tab=w8\">Play</a> <a class=gb1 href=\"https://www.youtube.com/?tab=w1\">YouTube</a> <a class=gb1 href=\"https://news.google.com/?tab=wn\">Nieuws</a> <a class=gb1 href=\"https://mail.google.com/mail/?tab=wm\">Gmail</a> <a class=gb1 href=\"https://drive.google.com/?tab=wo\">Drive</a> <a class=gb1 style=\"text-decoration:none\" href=\"https://www.google.nl/intl/nl/about/products?tab=wh\"><u>Meer</u> &raquo;</a></nobr></div><div id=guser width=100%><nobr><span id=gbn class=gbi></span><span id=gbf class=gbf></span><span id=gbe></span><a href=\"http://www.google.nl/history/optout?hl=nl\" class=gb4>Webgeschiedenis</a> | <a  href=\"/preferences?hl=nl\" class=gb4>Instellingen</a> | <a target=_top id=gb_70 href=\"https://accounts.google.com/ServiceLogin?hl=nl&passive=true&continue=https://www.google.com/&ec=GAZAAQ\" class=gb4>Inloggen</a></nobr></div><div class=gbh style=left:0></div><div class=gbh style=right:0></div></div><center><br clear=\"all\" id=\"lgpd\"><div id=\"lga\"><img alt=\"Google\" height=\"92\" src=\"/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.png\" style=\"padding:28px 0 14px\" width=\"272\" id=\"hplogo\"><br><br></div><form action=\"/search\" name=\"f\"><table cellpadding=\"0\" cellspacing=\"0\"><tr valign=\"top\"><td width=\"25%\">&nbsp;</td><td align=\"center\" nowrap=\"\"><input name=\"ie\" value=\"ISO-8859-1\" type=\"hidden\"><input value=\"nl\" name=\"hl\" type=\"hidden\"><input name=\"source\" type=\"hidden\" value=\"hp\"><input name=\"biw\" type=\"hidden\"><input name=\"bih\" type=\"hidden\"><div class=\"ds\" style=\"height:32px;margin:4px 0\"><input class=\"lst\" style=\"margin:0;padding:5px 8px 0 6px;vertical-align:top;color:#000\" autocomplete=\"off\" value=\"\" title=\"Google Zoeken\" maxlength=\"2048\" name=\"q\" size=\"57\"></div><br style=\"line-height:0\"><span class=\"ds\"><span class=\"lsbb\"><input class=\"lsb\" value=\"Google Zoeken\" name=\"btnG\" type=\"submit\"></span></span><span class=\"ds\"><span class=\"lsbb\"><input class=\"lsb\" id=\"tsuid_1\" value=\"Ik doe een gok\" name=\"btnI\" type=\"submit\"><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){var id='tsuid_1';document.getElementById(id).onclick = function(){if (this.form.q.value){this.checked = 1;if (this.form.iflsig)this.form.iflsig.disabled = false;}\n","else top.location='/doodles/';};})();</script><input value=\"AOEireoAAAAAZIW4O1LPYGgIZlEXkgwKyWOGMtw_-e47\" name=\"iflsig\" type=\"hidden\"></span></span></td><td class=\"fl sblc\" align=\"left\" nowrap=\"\" width=\"25%\"><a href=\"/advanced_search?hl=nl&amp;authuser=0\">Geavanceerd zoeken</a></td></tr></table><input id=\"gbv\" name=\"gbv\" type=\"hidden\" value=\"1\"><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){var a,b=\"1\";if(document&&document.getElementById)if(\"undefined\"!=typeof XMLHttpRequest)b=\"2\";else if(\"undefined\"!=typeof ActiveXObject){var c,d,e=[\"MSXML2.XMLHTTP.6.0\",\"MSXML2.XMLHTTP.3.0\",\"MSXML2.XMLHTTP\",\"Microsoft.XMLHTTP\"];for(c=0;d=e[c++];)try{new ActiveXObject(d),b=\"2\"}catch(h){}}a=b;if(\"2\"==a&&-1==location.search.indexOf(\"&gbv=2\")){var f=google.gbvu,g=document.getElementById(\"gbv\");g&&(g.value=a);f&&window.setTimeout(function(){location.href=f},0)};}).call(this);</script></form><div id=\"gac_scont\"></div><div style=\"font-size:83%;min-height:3.5em\"><br><div id=\"gws-output-pages-elements-homepage_additional_languages__als\"><style>#gws-output-pages-elements-homepage_additional_languages__als{font-size:small;margin-bottom:24px}#SIvCob{color:#3c4043;display:inline-block;line-height:28px;}#SIvCob a{padding:0 3px;}.H6sW5{display:inline-block;margin:0 2px;white-space:nowrap}.z4hgWe{display:inline-block;margin:0 2px}</style><div id=\"SIvCob\">Google aangeboden in:  <a href=\"https://www.google.com/setprefs?sig=0_gvLVbDrRbrHyiHh3vnRAqIW9GNk%3D&amp;hl=fy&amp;source=homepage&amp;sa=X&amp;ved=0ahUKEwjUouH-iLv_AhVu7LsIHRixAe0Q2ZgBCAU\">Frysk</a>  </div></div></div><span id=\"footer\"><div style=\"font-size:10pt\"><div style=\"margin:19px auto;text-align:center\" id=\"WqQANb\"><a href=\"/intl/nl/ads/\">Adverteren</a><a href=\"http://www.google.nl/intl/nl/services/\">Bedrijfsoplossingen</a><a href=\"/intl/nl/about.html\">Alles over Google</a><a href=\"https://www.google.com/setprefdomain?prefdom=NL&amp;prev=https://www.google.nl/&amp;sig=K_Z8CDjY6PHzrMVPnyKbRhy1g8aVQ%3D\">Google.nl</a></div></div><p style=\"font-size:8pt;color:#70757a\">&copy; 2023 - <a href=\"/intl/nl/policies/privacy/\">Privacy</a> - <a href=\"/intl/nl/policies/terms/\">Voorwaarden</a></p></span></center><script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){window.google.cdo={height:757,width:1440};(function(){var a=window.innerWidth,b=window.innerHeight;if(!a||!b){var c=window.document,d=\"CSS1Compat\"==c.compatMode?c.documentElement:c.body;a=d.clientWidth;b=d.clientHeight}\n","if(a&&b&&(a!=google.cdo.width||b!=google.cdo.height)){var e=google,f=e.log,g=\"/client_204?&atyp=i&biw=\"+a+\"&bih=\"+b+\"&ei=\"+google.kEI,h=\"\",k=[],l=void 0!==window.google&&void 0!==window.google.kOPI&&0!==window.google.kOPI?window.google.kOPI:null;null!=l&&k.push([\"opi\",l.toString()]);for(var m=0;m<k.length;m++){if(0===m||0<m)h+=\"&\";h+=k[m][0]+\"=\"+k[m][1]}f.call(e,\"\",\"\",g+h)};}).call(this);})();</script> <script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){google.xjs={ck:'xjs.hp.JFYUd30fzuo.L.X.O',cs:'ACT90oE08UuoGTABMqqsWzLeLG5-Cj4t9g',csss:'ACT90oGEwD0vLZu5vmP0xxtFnfKksvscew',excm:[],sepcss:false};})();</script>  <script nonce=\"3_EpQBjq8mGQU4I1P8Vazg\">(function(){var u='/xjs/_/js/k\\x3dxjs.hp.en.EVk_zeDfSws.O/am\\x3dAAAAAAAAAEAAAAAAAAAAAAAAQCcAgAIACw/d\\x3d1/ed\\x3d1/rs\\x3dACT90oGUwIW5Pl_nuJkZur8wj9syHnTWmg/m\\x3dsb_he,d,cEt90b,SNUn3,qddgKe,sTsDMc,dtl0hd,eHDfl';var amd=0;\n","var e=this||self,f=function(c){return c};var g;var k=function(c){this.g=c};k.prototype.toString=function(){return this.g+\"\"};var m={};\n","function q(){var c=u,n=function(){};google.lx=google.stvsc?n:function(){google.timers&&google.timers.load&&google.tick&&google.tick(\"load\",\"xjsls\");var a=document;var b=\"SCRIPT\";\"application/xhtml+xml\"===a.contentType&&(b=b.toLowerCase());b=a.createElement(b);a=null===c?\"null\":void 0===c?\"undefined\":c;if(void 0===g){var d=null;var l=e.trustedTypes;if(l&&l.createPolicy){try{d=l.createPolicy(\"goog#html\",{createHTML:f,createScript:f,createScriptURL:f})}catch(r){e.console&&e.console.error(r.message)}g=\n","d}else g=d}a=(d=g)?d.createScriptURL(a):a;a=new k(a,m);b.src=a instanceof k&&a.constructor===k?a.g:\"type_error:TrustedResourceUrl\";var h,p;(h=(a=null==(p=(h=(b.ownerDocument&&b.ownerDocument.defaultView||window).document).querySelector)?void 0:p.call(h,\"script[nonce]\"))?a.nonce||a.getAttribute(\"nonce\")||\"\":\"\")&&b.setAttribute(\"nonce\",h);document.body.appendChild(b);google.psa=!0;google.lx=n};google.bx||google.lx()};google.xjsu=u;e._F_jsUrl=u;setTimeout(function(){0<amd?google.caft(function(){return q()},amd):q()},0);})();window._ = window._ || {};window._DumpException = _._DumpException = function(e){throw e;};window._s = window._s || {};_s._DumpException = _._DumpException;window._qs = window._qs || {};_qs._DumpException = _._DumpException;function _F_installCss(c){}\n","(function(){google.jl={blt:'none',chnk:0,dw:false,dwu:true,emtn:0,end:0,ico:false,ikb:0,ine:false,injs:'none',injt:0,injth:0,injv2:false,lls:'default',pdt:0,rep:0,snet:true,strt:0,ubm:false,uwp:true};})();(function(){var pmc='{\\x22d\\x22:{},\\x22sb_he\\x22:{\\x22agen\\x22:true,\\x22cgen\\x22:true,\\x22client\\x22:\\x22heirloom-hp\\x22,\\x22dh\\x22:true,\\x22ds\\x22:\\x22\\x22,\\x22fl\\x22:true,\\x22host\\x22:\\x22google.com\\x22,\\x22jsonp\\x22:true,\\x22msgs\\x22:{\\x22cibl\\x22:\\x22Zoekopdracht wissen\\x22,\\x22dym\\x22:\\x22Bedoelde je:\\x22,\\x22lcky\\x22:\\x22Ik doe een gok\\x22,\\x22lml\\x22:\\x22Meer informatie\\x22,\\x22psrc\\x22:\\x22Deze zoekopdracht is verwijderd uit je \\\\u003Ca href\\x3d\\\\\\x22/history\\\\\\x22\\\\u003EWebgeschiedenis\\\\u003C/a\\\\u003E\\x22,\\x22psrl\\x22:\\x22Verwijderen\\x22,\\x22sbit\\x22:\\x22Zoeken op afbeelding\\x22,\\x22srch\\x22:\\x22Google Zoeken\\x22},\\x22ovr\\x22:{},\\x22pq\\x22:\\x22\\x22,\\x22rfs\\x22:[],\\x22sbas\\x22:\\x220 3px 8px 0 rgba(0,0,0,0.2),0 0 0 1px rgba(0,0,0,0.08)\\x22,\\x22stok\\x22:\\x22SpIBRsBNAtOypO5JfGzHKfpAmcw\\x22}}';google.pmc=JSON.parse(pmc);})();(function(){\n","var b=function(a){var c=0;return function(){return c<a.length?{done:!1,value:a[c++]}:{done:!0}}},e=this||self;var g,h;a:{for(var k=[\"CLOSURE_FLAGS\"],l=e,n=0;n<k.length;n++)if(l=l[k[n]],null==l){h=null;break a}h=l}var p=h&&h[610401301];g=null!=p?p:!1;var q,r=e.navigator;q=r?r.userAgentData||null:null;function t(a){return g?q?q.brands.some(function(c){return(c=c.brand)&&-1!=c.indexOf(a)}):!1:!1}function u(a){var c;a:{if(c=e.navigator)if(c=c.userAgent)break a;c=\"\"}return-1!=c.indexOf(a)};function v(){return g?!!q&&0<q.brands.length:!1}function w(){return u(\"Safari\")&&!(x()||(v()?0:u(\"Coast\"))||(v()?0:u(\"Opera\"))||(v()?0:u(\"Edge\"))||(v()?t(\"Microsoft Edge\"):u(\"Edg/\"))||(v()?t(\"Opera\"):u(\"OPR\"))||u(\"Firefox\")||u(\"FxiOS\")||u(\"Silk\")||u(\"Android\"))}function x(){return v()?t(\"Chromium\"):(u(\"Chrome\")||u(\"CriOS\"))&&!(v()?0:u(\"Edge\"))||u(\"Silk\")}function y(){return u(\"Android\")&&!(x()||u(\"Firefox\")||u(\"FxiOS\")||(v()?0:u(\"Opera\"))||u(\"Silk\"))};var z=v()?!1:u(\"Trident\")||u(\"MSIE\");y();x();w();var A=!z&&!w(),D=function(a){if(/-[a-z]/.test(\"ved\"))return null;if(A&&a.dataset){if(y()&&!(\"ved\"in a.dataset))return null;a=a.dataset.ved;return void 0===a?null:a}return a.getAttribute(\"data-\"+\"ved\".replace(/([A-Z])/g,\"-$1\").toLowerCase())};var E=[],F=null;function G(a){a=a.target;var c=performance.now(),f=[],H=f.concat,d=E;if(!(d instanceof Array)){var m=\"undefined\"!=typeof Symbol&&Symbol.iterator&&d[Symbol.iterator];if(m)d=m.call(d);else if(\"number\"==typeof d.length)d={next:b(d)};else throw Error(\"a`\"+String(d));for(var B=[];!(m=d.next()).done;)B.push(m.value);d=B}E=H.call(f,d,[c]);if(a&&a instanceof HTMLElement)if(a===F){if(c=4<=E.length)c=5>(E[E.length-1]-E[E.length-4])/1E3;if(c){c=google.getEI(a);a.hasAttribute(\"data-ved\")?f=a?D(a)||\"\":\"\":f=(f=\n","a.closest(\"[data-ved]\"))?D(f)||\"\":\"\";f=f||\"\";if(a.hasAttribute(\"jsname\"))a=a.getAttribute(\"jsname\");else{var C;a=null==(C=a.closest(\"[jsname]\"))?void 0:C.getAttribute(\"jsname\")}google.log(\"rcm\",\"&ei=\"+c+\"&ved=\"+f+\"&jsname=\"+(a||\"\"))}}else F=a,E=[c]}window.document.addEventListener(\"DOMContentLoaded\",function(){document.body.addEventListener(\"click\",G)});}).call(this);</script></body></html>\n"]}]},{"cell_type":"markdown","source":["위 예시에서는, [google.com](google.com)의 HTML 구조를 가져옵니다. 이는 해당 페이지에서 Ctrl+U 키나 F12키를 눌러 개발자 모드로 들어가면 확인할 수 있습니다."],"metadata":{"id":"6U-Kmr_JJb6G"}},{"cell_type":"markdown","source":["### BeautifulSoup 이용하기\n","\n","위의 request 라이브러리를 이용하여 웹의 HTML 구조를 받아올 수 있습니다. 이를 통해 네이버 기사의 제목을 가져오는 간단한 프로그램을 만들어 보겠습니다.\n","\n","[네이버 뉴스에서 고양이를 검색한 링크](https://search.naver.com/search.naver?where=news&sm=tab_jum&query=%EA%B3%A0%EC%96%91%EC%9D%B4)를 이용하겠습니다."],"metadata":{"id":"F60LZSLpJ6I9"}},{"cell_type":"code","source":["from bs4 import BeautifulSoup\n","import requests\n","\n","url = \"https://search.naver.com/search.naver?where=news&sm=tab_jum&query=%EA%B3%A0%EC%96%91%EC%9D%B4\"\n","\n","# requests 모듈을 이용하여 GET 요청\n","response = requests.get(url)\n","\n","#  html 코드와 사용할 파서를 지정\n","soup = BeautifulSoup(response.text, \"html.parser\")"],"metadata":{"id":"VLli_CIUKiEo"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["위 링크에서 F12를 통해 개발자 모드로 들어가보면,   \n","<img src=\"https://github.com/sleepncaffeine/my_small_python_notes/assets/101965838/04180da3-5634-4a95-a68a-bbef0ea6c55a\" width=\"650px\"></img>   \n","찾으려는 뉴스 기사의 제목이 `a.news_tit`선택자를 가짐을 알 수 있습니다."],"metadata":{"id":"_q7z-8dtK3VC"}},{"cell_type":"code","source":["# select_one으로 맨 앞의 하나 선택\n","print(soup.select_one(\"a.news_tit\").get_text())"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WhLgE3vrK27h","executionInfo":{"status":"ok","timestamp":1686482318837,"user_tz":-540,"elapsed":516,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"239dc7e1-aca8-421f-f1a6-0bfba887ff05"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["고양이 한마리 겨우 빠져나갈 지구대 창문으로 외국인 10명 도주\n"]}]},{"cell_type":"code","source":["# select로 리스트 선택\n","titles = soup.select(\"a.news_tit\")\n","\n","for i in titles:\n","    title = i.get_text()\n","    print(title)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"jlVxQOiZLjFS","executionInfo":{"status":"ok","timestamp":1686482350423,"user_tz":-540,"elapsed":3,"user":{"displayName":"박진재","userId":"13496824967263428254"}},"outputId":"20f81b95-fe29-4076-851c-dcf0c6e4f02a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["고양이 한마리 겨우 빠져나갈 지구대 창문으로 외국인 10명 도주\n","길고양이, ‘동네 고양이’라 부르고 해법 찾아봐요\n","시민단체 \"개·고양이 보신문화는 악습…칠성 개시장 폐쇄하라\"\n","개ㆍ고양이 수십 마리 펫숍에 버리고 잠적한 일당 검거\n","강남 빌라 담장 올라 20분간 고양이 봤다는 남성…법원 \"건물침입\" 벌금형\n","송영길 “김건희 소환은?…檢, 고양이 앞의 쥐 모양새”\n","[고양이 눈]“아빠 구해주세요∼!”\n","송영길 \"검찰, 김건희 앞에선 고양이 앞에 쥐…주변 말고 나를 소환해라\"\n","치어리더 정가예 '귀여운 고양이 스타일'[엑's HD포토]\n","새끼 사체 발견된 고양이카페 '영업정지 15일'...솜방망이 처벌 논란\n"]}]}]}