Description
Hi @urialon.
Could you please tell me how do you construct a json for the service https://po3g2dx2qa.execute-api.us-east-1.amazonaws.com/production/extractmethods
?
I requested the following procedure (you used it in the previous issue):
public boolean f(Set<String> set, String value) { for (String entry : set) { if (entry.equalsIgnoreCase(value)) { return true ; } } return false; }
I tried to run JavaExtractor with the following params (I found them in extract.py
): Running: java -cp /app/JavaExtractor/JPredict/target/JavaExtractor-0.0.2-SNAPSHOT.jar JavaExtractor.App --max_path_length 9 --max_path_width 2 --file /app/Input.java
.
It returned me the following result:
f boolean,Prim0|Mth|Nm1,METHOD_NAME boolean,Prim0|Mth|Prm|VDID0,set boolean,Prim0|Mth|Prm|Cls|Cls0,string METHOD_NAME,Nm1|Mth|Prm|VDID0,set METHOD_NAME,Nm1|Mth|Prm|Cls|Cls0,string METHOD_NAME,Nm1|Mth|Prm|VDID0,value METHOD_NAME,Nm1|Mth|Prm|Cls1,string set,VDID0|Prm|Cls|Cls0,string set,VDID0|Prm|Mth|Prm|VDID0,value set,VDID0|Prm|Mth|Prm|Cls1,string set,VDID0|Prm|Mth|Bk|Foreach|VDE|Cls0,string set,VDID0|Prm|Mth|Bk|Foreach|VDE|VD|VDID0,entry set,VDID0|Prm|Mth|Bk|Foreach|Nm1,set set,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0,entry set,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2,value set,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case set,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true set,VDID0|Prm|Mth|Bk|Ret|BoolEx0,false string,Cls0|Cls|Prm|Mth|Prm|VDID0,value string,Cls0|Cls|Prm|Mth|Prm|Cls1,string string,Cls0|Cls|Prm|Mth|Bk|Foreach|VDE|Cls0,string string,Cls0|Cls|Prm|Mth|Bk|Foreach|VDE|VD|VDID0,entry string,Cls0|Cls|Prm|Mth|Bk|Foreach|Nm1,set string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0,entry string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2,value string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case string,Cls0|Cls|Prm|Mth|Bk|Ret|BoolEx0,false value,VDID0|Prm|Cls1,string value,VDID0|Prm|Mth|Bk|Foreach|VDE|Cls0,string value,VDID0|Prm|Mth|Bk|Foreach|VDE|VD|VDID0,entry value,VDID0|Prm|Mth|Bk|Foreach|Nm1,set value,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0,entry value,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2,value value,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case value,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true value,VDID0|Prm|Mth|Bk|Ret|BoolEx0,false string,Cls1|Prm|Mth|Bk|Foreach|VDE|Cls0,string string,Cls1|Prm|Mth|Bk|Foreach|VDE|VD|VDID0,entry string,Cls1|Prm|Mth|Bk|Foreach|Nm1,set string,Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0,entry string,Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2,value string,Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case string,Cls1|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true string,Cls1|Prm|Mth|Bk|Ret|BoolEx0,false string,Cls0|VDE|VD|VDID0,entry string,Cls0|VDE|Foreach|Nm1,set string,Cls0|VDE|Foreach|Bk|If|Cal0|Nm0,entry string,Cls0|VDE|Foreach|Bk|If|Cal0|Nm2,value string,Cls0|VDE|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case string,Cls0|VDE|Foreach|Bk|If|Bk|Ret|BoolEx0,true string,Cls0|VDE|Foreach|Bk|Ret|BoolEx0,false entry,VDID0|VD|VDE|Foreach|Nm1,set entry,VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm0,entry entry,VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm2,value entry,VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case entry,VDID0|VD|VDE|Foreach|Bk|If|Bk|Ret|BoolEx0,true entry,VDID0|VD|VDE|Foreach|Bk|Ret|BoolEx0,false set,Nm1|Foreach|Bk|If|Cal0|Nm0,entry set,Nm1|Foreach|Bk|If|Cal0|Nm2,value set,Nm1|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case set,Nm1|Foreach|Bk|If|Bk|Ret|BoolEx0,true set,Nm1|Foreach|Bk|Ret|BoolEx0,false entry,Nm0|Cal|Nm2,value entry,Nm0|Cal|If|Bk|Ret|BoolEx0,true entry,Nm0|Cal|If|Bk|Foreach|Bk|Ret|BoolEx0,false value,Nm2|Cal|Nm3,equals|ignore|case value,Nm2|Cal|If|Bk|Ret|BoolEx0,true value,Nm2|Cal|If|Bk|Foreach|Bk|Ret|BoolEx0,false equals|ignore|case,Nm3|Cal|If|Bk|Ret|BoolEx0,true equals|ignore|case,Nm3|Cal|If|Bk|Foreach|Bk|Ret|BoolEx0,false true,BoolEx0|Ret|Bk|If|Bk|Foreach|Bk|Ret|BoolEx0,false\n'
I noticed that some values are missed, e.g.:
"name1": "boolean",
"name2": "set",
"shortPath": "Prim0|Mth|Prm|GenericClass1",
"path": "(PrimitiveType0)^(MethodDeclaration)_(Parameter)_(GenericClass1)",
"name1NodeId": 2,
"name2NodeId": 6,
"name1TokenNum": 0,
"name2TokenNum": 0
How can I get absolutely the same items?
-
Actually, I see that "name1NodeId", "name2NodeId", "name1TokenNum", "name2TokenNum" are pointed to the nodes of the ast tree and they are not needed in the prediction function. Also, I see that ast tree is not also used. Is it right?
-
So, the last part is to understand how you are getting the response.
Usually there are missing values in the result of JavaExtractor, and they are usually related toGenericClass1
value.
Also, I noticed that some values in the response are missed, but they exist in the result of JavaExtractor:
{'value,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|VDE|VD|VDID0,entry', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|VDE|Cls0,string', 'string,Cls0|Cls|Prm|Mth|Prm|Cls1,string', 'set,VDID0|Prm|Cls|Cls0,string', 'string,Cls0|Cls|Prm|Mth|Prm|VDID0,value', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0,entry', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2,value', 'METHOD_NAME,Nm1|Mth|Prm|Cls|Cls0,string', 'set,VDID0|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true', 'true,BoolEx0|Ret|Bk|If|Bk|Foreach|Bk|Ret|BoolEx0,false\\n', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|Nm1,set', 'string,Cls0|Cls|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3,equals|ignore|case', 'string,Cls0|Cls|Prm|Mth|Bk|Ret|BoolEx0,false', 'string,Cls1|Prm|Mth|Bk|Foreach|Bk|If|Bk|Ret|BoolEx0,true', 'boolean,Prim0|Mth|Prm|Cls|Cls0,string'}
Could you please tell me, how do u run JavaExtractor on your server?
Activity
urialon commentedon Oct 26, 2021
Hi @lyriccoder ,
Can you please remind me what are you trying to achieve?
You are right that
name1NodeId
etc are not needed in the prediction, they are for visualization purposes on the code2seq.org website.So why do you need them? Or which fields do you need?
lyriccoder commentedon Oct 27, 2021
Currently, Code2seq sends a request to a server. I want to run it locally, without any network. I need to get the same json as your service returns.
Since I can run JavaExtractor manually, I want to run it locally, get the results and create json which is passed to the prediction function.
But I see that the results of JavaExtractor and your service are different.
I need to know the way you run JavaExtractor on your server, maybe you have additional parameters, etc. I just want to create the same json you return from the AWS server.
For some reason, your server returns more data: Some nodes are included in your response, but not included when I run JavaExtractor. E.g., I noticed if the
shortPath
field of the json contains theGenericClass
substring, it won't be included in the result of JavaExtrator.urialon commentedon Oct 28, 2021
Hi @lyriccoder ,
So my question is why do you need exactly the same json?
Some fields there are used only for visualization on the website.
lyriccoder commentedon Oct 28, 2021
I meant the json with fields which are needed for prediction.
Some nodes are missing... Seems you are using another parameters in JavaExtractor. It can affect the prediction.
urialon commentedon Oct 28, 2021
lyriccoder commentedon Oct 29, 2021
I did it in the previous message. Ok, let me show it again.
I get the following response:
Http Response
java -cp JavaExtractor/JPredict/target/JavaExtractor-0.0.2-SNAPSHOT.jar JavaExtractor.App --max_path_length 9 --max_path_width 2 --file Input.java
I get the following response:
Response of JavaExtractor
As you see, the examples with
GenericClass
is only presented here4) Inverted difference is demonstrated below. Those values are in the result of JavaExtractor, but they are missed in the http response:
urialon commentedon Nov 2, 2021
I see. Good catch!
I will try to debug this. In the meantime, you can use the JavaExtractor because this is the extractor that the model was trained with. The cases of "inverted difference" might be due to the differences in parameters: you used
--max_path_length 9
instead of--max_path_length 8
.Uri
AUZQ commentedon Dec 3, 2021
Hi@lyriccoder
Could you please tell me how do you run JavaExtractor locally ?
I found your repository code2seq still used the api to extract codes, but in this issue it seemed that you had already run it locally.My purpose is to extract codes locally.
lyriccoder commentedon Dec 3, 2021
java -cp JavaExtractor/JPredict/target/JavaExtractor-0.0.2-SNAPSHOT.jar JavaExtractor.App --max_path_length=8 --max_path_width=2 --file filename
There is a hardcoded filename inside
extractor.py
orinteractive_predict.py
. It rewritesInput.java
and passes into thefilename
paramAUZQ commentedon Dec 3, 2021
Thanks!!!! I made it!