-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
159 lines (148 loc) · 12.3 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta charset="UTF-8">
<title>Treehuggerdocs by autosoft-dev</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" type="text/css" href="stylesheets/normalize.css" media="screen">
<link href='https://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
<link rel="stylesheet" type="text/css" href="stylesheets/stylesheet.css" media="screen">
<link rel="stylesheet" type="text/css" href="stylesheets/github-light.css" media="screen">
</head>
<body>
<section class="page-header">
<h1 class="project-name">Treehuggerdocs</h1>
<h2 class="project-tagline">The documentation website source for tree-hugger</h2>
<a href="https://github.com/autosoft-dev/treehuggerdocs" class="btn">View on GitHub</a>
<a href="https://github.com/autosoft-dev/treehuggerdocs/zipball/master" class="btn">Download .zip</a>
<a href="https://github.com/autosoft-dev/treehuggerdocs/tarball/master" class="btn">Download .tar.gz</a>
</section>
<section class="main-content">
<h1>
<a id="welcome-to-tree-hugger" class="anchor" href="#welcome-to-tree-hugger" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Welcome to tree-hugger</h1>
<p>A light-weight, high level, universal code parser built on top of tree-sitter</p>
<h1>
<a id="browse-the-documentation" class="anchor" href="#browse-the-documentation" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Browse the documentation</h1>
<h2>
<a id="installation-and-setup" class="anchor" href="#installation-and-setup" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation and setup</h2>
<h3>
<a id="from-pip" class="anchor" href="#from-pip" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>From pip:</h3>
<p>Just do</p>
<pre><code>pip install tree-hugger
</code></pre>
<h3>
<a id="from-source" class="anchor" href="#from-source" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>From Source:</h3>
<pre><code>git clone https://github.com/autosoft-dev/tree-hugger.git
cd tree-hugger
pip install -e .
</code></pre>
<p><em>The installation process is tested in macOS Mojave, we have a <a href="https://github.com/autosoft-dev/tree-sitter-docker">separate docker binding</a> for compiling the libraries for Linux and soon this library will be integrated in that as well</em></p>
<p><em>You may need to install libgit2. In case you are in mac just use <code>brew install libgit2</code></em></p>
<h3>
<a id="building-the-so-files" class="anchor" href="#building-the-so-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building the .so files</h3>
<p><em>Please note that building the libraries has been tested under a macOS Mojave with Apple LLVM version 10.0.1 (clang-1001.0.46.4)</em></p>
<p><em>Please check out our Linux specific instructions <a href="https://github.com/autosoft-dev/tree-sitter-docker">here</a></em></p>
<p>Once this library is installed it gives you a command line utility to download and compile tree-sitter .so files with ease. As an example -</p>
<pre><code>create_libs python
</code></pre>
<p>Here is the full usage guide of the command</p>
<pre><code>usage: create_libs [-h] [-c] [-l LIB_NAME] langs [langs ...]
positional arguments:
langs Give the name of languages for tree-sitter (php,
python, go ...)
optional arguments:
-h, --help show this help message and exit
-c, --copy-to-workspace
Shall we copy the created libs to the present dir?
(default: False)
-l LIB_NAME, --lib-name LIB_NAME
The name of the generated .so file
</code></pre>
<h2>
<a id="supported-languages" class="anchor" href="#supported-languages" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Supported languages</h2>
<p>At the moment, Tree-Hugger supports the following languages:</p>
<ul>
<li>Python</li>
<li>PHP</li>
</ul>
<p>Because Tree-Hugger is easily expandable, the users can extends the functionality with a custom parser or implement a new one for another language</p>
<h2>
<a id="tutorials" class="anchor" href="#tutorials" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Tutorials</h2>
<h3>
<a id="a-quick-example" class="anchor" href="#a-quick-example" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>A Quick Example</h3>
<p>First run the above command to generate the libraries.</p>
<p>In our settings we just use the <code>-c</code> flag to copy the generated <code>tree-sitter</code> library's <code>.so</code> file to our workspace.
And once copied, we place it under a directory called <code>tslibs</code> (It is in the .gitignore). But of course, if you are using linux then this command probably won't work and you will need to use our <a href="https://github.com/autosoft-dev/tree-sitter-docker">tree-sitter-docker</a> image and manually copy the final .so file.</p>
<p>Another thing that we need before we can analyze any code file is an yaml with queries. We have suuplied one example query file
under <a href="https://raw.githubusercontent.com/autosoft-dev/tree-hugger/master/queries/example_queries.yml"><strong>queries</strong></a> directory.</p>
<p><em>Please note that, you can set up two environment variables <code>QUERY_FILE_PATH</code> and <code>TS_LIB_PATH</code> for the query file path and
tree-sitter lib path and then the libary will use them automatically. Otherwise, as an alternative, you can pass it when creating any <code>*Parser</code> object</em></p>
<p>Assuming that you have the necessary environment variable setup. The following line of code will create a <code>PythonParser</code> object</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">tree_hugger</span>.<span class="pl-s1">core</span> <span class="pl-k">import</span> <span class="pl-v">PythonParser</span>
<span class="pl-s1">pp</span> <span class="pl-c1">=</span> <span class="pl-v">PythonParser</span>()</pre></div>
<p>And then you can pass in any Python file that you want to analyze, like so :</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">pp</span>.<span class="pl-en">parse_file</span>(<span class="pl-s">"tests/assets/file_with_different_functions.py"</span>)
<span class="pl-v">Out</span>[<span class="pl-c1">3</span>]: <span class="pl-c1">True</span></pre></div>
<p><code>parse_file</code> returns <code>True</code> if success</p>
<p>And then you are free to use the methods exposed by that particular Parser object. As an example -</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">pp</span>.<span class="pl-en">get_all_function_names</span>()
<span class="pl-v">Out</span>[<span class="pl-c1">4</span>]:
[<span class="pl-s">'first_child'</span>,
<span class="pl-s">'second_child'</span>,
<span class="pl-s">'say_whee'</span>,
<span class="pl-s">'wrapper'</span>,
<span class="pl-s">'my_decorator'</span>,
<span class="pl-s">'parent'</span>]</pre></div>
<p>OR</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">pp</span>.<span class="pl-en">get_all_function_documentations</span>()
<span class="pl-v">Out</span>[<span class="pl-c1">5</span>]:
{<span class="pl-s">'parent'</span>: <span class="pl-s">'"""This is the parent function<span class="pl-cce">\n</span> <span class="pl-cce">\n</span> There are other lines in the doc string<span class="pl-cce">\n</span> This is the third line<span class="pl-cce">\n</span><span class="pl-cce">\n</span> And this is the fourth<span class="pl-cce">\n</span> """'</span>,
<span class="pl-s">'first_child'</span>: <span class="pl-s">"'''<span class="pl-cce">\n</span> This is first child<span class="pl-cce">\n</span> '''"</span>,
<span class="pl-s">'second_child'</span>: <span class="pl-s">'"""<span class="pl-cce">\n</span> This is second child<span class="pl-cce">\n</span> """'</span>,
<span class="pl-s">'my_decorator'</span>: <span class="pl-s">'"""<span class="pl-cce">\n</span> Outer decorator function<span class="pl-cce">\n</span> """'</span>,
<span class="pl-s">'say_whee'</span>: <span class="pl-s">'"""<span class="pl-cce">\n</span> Hellooooooooo<span class="pl-cce">\n</span><span class="pl-cce">\n</span> This is a function with decorators<span class="pl-cce">\n</span> """'</span>}</pre></div>
<h2>
<a id="api-reference" class="anchor" href="#api-reference" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>API reference</h2>
<h3>
<a id="queries" class="anchor" href="#queries" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Queries:</h3>
<p>Queries are s-expressions (Remember LISP?) that works on the parsed code and gives you what you want. They are a great way to fetch arbitary data from the parsed code without having to travel through it recursively.
Tree-hugger gives you a way to write your queries in yaml file (Check out the <code>queries.yml</code> files in the parsers file to see some examples).</p>
<p>This main section is further sub-divded into few (as many as you need, actually) sections. Each of them has the same structure. A name of a query followed by the query itself. Written as an s-expression. One example:</p>
<pre><code>all_function_docstrings:
"
(
function_definition
name: (identifier) @function.def
body: (block(expression_statement(string))) @function.docstring
)
"
</code></pre>
<p>Of course, you have to follow yaml grammar while writing these queries. You can see a bit more about writng these queries in the documentation of tree-sitter. <a href="https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries">Here</a>. Although it is not very intuitive to start with. We are planning to write a detailed tutorial on this subject.</p>
<h3>
<a id="parser-class" class="anchor" href="#parser-class" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Parser Class:</h3>
<p>A parser class extends from the BaseParser class. The only mandatory argument that a Parser class should pass to the parent is the <code>language</code>. This is a string. Such as <code>python</code> (remember, lower case). Although, each parser class must
have the options to take in the path of the tree-sitter library (.so file that we are using to parse the code) and the path to the queries yaml file, in their constructor.</p>
<p>The BaseParser class can do few things for you.</p>
<ul>
<li>
<p>It loads and prepares the .so file with respect to the language you just mentioned.</p>
</li>
<li>
<p>It loads, parses, and prepares the query yaml file. (for the queries, we internally use an extended UserDict class. More on that later.)</p>
</li>
<li>
<p>It gives an API to parse a file and prepare it for query. <code>BaseParser.parse_file</code></p>
</li>
<li>
<p>It also gives you another (most likely not to be exposed outside) API <code>_run_query_and_get_captures</code> which lets you run any queries and return back the matched results (if any) from the parsed tree.</p>
</li>
</ul>
<p>If you are interested to see the example of one of the methods in the PythonParser class, to know how all of these come together. Here it is (Do not forget, we use those APIs once we have called <code>parse_file</code> and parsed the file) -</p>
<p>The function <code>match_from_span</code> is a very handy function. It is defined in the BaseParser module. It takes a span definition and returns the underlying code string from it.</p>
<footer class="site-footer">
<span class="site-footer-owner"><a href="https://github.com/autosoft-dev/treehuggerdocs">Treehuggerdocs</a> is maintained by <a href="https://github.com/autosoft-dev">autosoft-dev</a>.</span>
<span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a> using the <a href="https://github.com/jasonlong/cayman-theme">Cayman theme</a> by <a href="https://twitter.com/jasonlong">Jason Long</a>.</span>
</footer>
</section>
</body>
</html>