4230 capture user queries #4470
Conversation
Capture queries for further analysis in order to improve search algorithm Changes made in search to models, views, generated migrations as per the guidelines Fixes: #4230
Capture queries for further analysis in order to improve search algorithm Changes made in search to models, views, generated migrations as per the guidelines Fixes: #4230
for more information, see https://pre-commit.ci
|
Looks like you added your entire venv. Definitely recommend creating a new branch without these changes. It's not enough to just create another commit removing the files because we won't want to commit them and ship them around as part of the repo forever. |
![semgrep-app[bot]](https://avatars.githubusercontent.com/in/60555?s=60&v=4){kind=small}
| DictCache(), cache_etags=True, serializer=None, heuristic=None | ||
| ) | ||
| sess = requests.Session() | ||
| sess.mount("http://", adapter) |
There was a problem hiding this comment.
Detected a request using 'http://'. This request will be unencrypted. Use 'https://' instead.
View Dataflow Graph
flowchart LR
classDef invis fill:white, stroke: none
classDef default fill:#e7f5ff, color:#1c7fd6, stroke: none
subgraph File0["<b>docker/courtlistener/myenv/lib/python3.12/site-packages/pip/_vendor/cachecontrol/_cmd.py</b>"]
direction LR
%% Source
subgraph Source
direction LR
v0["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/docker/courtlistener/myenv/lib/python3.12/site-packages/pip/_vendor/cachecontrol/_cmd.py#L33 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 33] "http://"</a>"]
end
%% Intermediate
%% Sink
subgraph Sink
direction LR
v1["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/docker/courtlistener/myenv/lib/python3.12/site-packages/pip/_vendor/cachecontrol/_cmd.py#L33 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 33] "http://"</a>"]
end
end
%% Class Assignment
Source:::invis
Sink:::invis
File0:::invis
%% Connections
Source --> Sink
| elif shell == "TerminalInteractiveShell": | ||
| return False # Terminal running IPython | ||
| else: | ||
| return False # Other type (?) |
There was a problem hiding this comment.
| return add + text + sub | ||
| return '<'+add+'>'+text+'</'+sub+'>' |
There was a problem hiding this comment.
code after return statement will not be executed
Ignore this finding from code-after-unconditional-return.
| def _verify_peercerts_impl( | ||
| ssl_context: ssl.SSLContext, | ||
| cert_chain: list[bytes], | ||
| server_hostname: str | None = None, | ||
| ) -> None: | ||
| # This is a no-op because we've enabled SSLContext's built-in | ||
| # verification via verify_mode=CERT_REQUIRED, and don't need to repeat it. | ||
| pass |
There was a problem hiding this comment.
pass is the body of function _verify_peercerts_impl. Consider removing this or raise NotImplementedError() if this is a TODO
| @typing.overload | ||
| def open( | ||
| file: Union[str, "PathLike[str]", bytes], | ||
| mode: Literal["rb"], | ||
| buffering: int = -1, | ||
| encoding: Optional[str] = None, | ||
| errors: Optional[str] = None, | ||
| newline: Optional[str] = None, | ||
| *, | ||
| total: Optional[int] = None, | ||
| description: str = "Reading...", | ||
| auto_refresh: bool = True, | ||
| console: Optional[Console] = None, | ||
| transient: bool = False, | ||
| get_time: Optional[Callable[[], float]] = None, | ||
| refresh_per_second: float = 10, | ||
| style: StyleType = "bar.back", | ||
| complete_style: StyleType = "bar.complete", | ||
| finished_style: StyleType = "bar.finished", | ||
| pulse_style: StyleType = "bar.pulse", | ||
| disable: bool = False, | ||
| ) -> ContextManager[BinaryIO]: | ||
| pass |
There was a problem hiding this comment.
pass is the body of function open. Consider removing this or raise NotImplementedError() if this is a TODO
| @typing.overload | ||
| def open( | ||
| file: Union[str, "PathLike[str]", bytes], | ||
| mode: Union[Literal["rt"], Literal["r"]], | ||
| buffering: int = -1, | ||
| encoding: Optional[str] = None, | ||
| errors: Optional[str] = None, | ||
| newline: Optional[str] = None, | ||
| *, | ||
| total: Optional[int] = None, | ||
| description: str = "Reading...", | ||
| auto_refresh: bool = True, | ||
| console: Optional[Console] = None, | ||
| transient: bool = False, | ||
| get_time: Optional[Callable[[], float]] = None, | ||
| refresh_per_second: float = 10, | ||
| style: StyleType = "bar.back", | ||
| complete_style: StyleType = "bar.complete", | ||
| finished_style: StyleType = "bar.finished", | ||
| pulse_style: StyleType = "bar.pulse", | ||
| disable: bool = False, | ||
| ) -> ContextManager[TextIO]: | ||
| pass |
There was a problem hiding this comment.
pass is the body of function open. Consider removing this or raise NotImplementedError() if this is a TODO
| def _onerror_ignore(*_args: Any) -> None: | ||
| pass |
There was a problem hiding this comment.
pass is the body of function _onerror_ignore. Consider removing this or raise NotImplementedError() if this is a TODO
| s += time.ctime().encode("utf-8") | ||
| s += os.urandom(8) | ||
|
|
||
| cnonce = hashlib.sha1(s).hexdigest()[:16] |
There was a problem hiding this comment.
Detected SHA1 hash algorithm which is considered insecure. SHA1 is not collision resistant and is therefore not suitable as a cryptographic signature. Use SHA256 or SHA3 instead.
Ignore this finding from insecure-hash-algorithm-sha1.
| def sha_utf8(x): | ||
| if isinstance(x, str): | ||
| x = x.encode("utf-8") | ||
| return hashlib.sha1(x).hexdigest() |
There was a problem hiding this comment.
Detected SHA1 hash algorithm which is considered insecure. SHA1 is not collision resistant and is therefore not suitable as a cryptographic signature. Use SHA256 or SHA3 instead.
Ignore this finding from insecure-hash-algorithm-sha1.
venv/bin/get_objgraph
Outdated
| try: | ||
| obj = objects[objtype] | ||
| except KeyError: | ||
| obj = pickle.load(open(objtype,'rb')) |
There was a problem hiding this comment.
Avoid using pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format.
venv/bin/undill
Outdated
| import sys | ||
| import dill | ||
| for file in sys.argv[1:]: | ||
| print (dill.load(open(file,'rb'))) |
There was a problem hiding this comment.
Avoid using dill, which uses pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format.
venv/bin/get_objgraph
Outdated
| try: | ||
| obj = objects[objtype] | ||
| except KeyError: | ||
| obj = pickle.load(open(objtype,'rb')) |
There was a problem hiding this comment.
Avoid using dill, which uses pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format.
venv/bin/get_objgraph
Outdated
|
|
||
| # get all objects for testing | ||
| from dill import load_types | ||
| load_types(pickleable=True,unpickleable=True) |
There was a problem hiding this comment.
Avoid using dill, which uses pickle, which is known to lead to code execution vulnerabilities. When unpickling, the serialized data could be manipulated to run arbitrary code. Instead, consider serializing the relevant data as JSON or a similar text-based serialization format.
| "ssl_version": self.protocol, | ||
| "server_side": server_side, | ||
| } | ||
| return wrap_socket(socket, ciphers=self.ciphers, **kwargs) |
There was a problem hiding this comment.
'ssl.wrap_socket()' is deprecated. This function creates an insecure socket without server name indication or hostname matching. Instead, create an SSL context using 'ssl.SSLContext()' and use that to wrap a socket.
Ignore this finding from ssl-wrap-socket-is-deprecated.
| new_style._link_id = style._link_id or self._link_id | ||
| new_style._null = style._null | ||
| if self._meta and style._meta: | ||
| new_style._meta = dumps({**self.meta, **style.meta}) |
There was a problem hiding this comment.
The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. See more details: https://docs.python.org/3/library/marshal.html?highlight=security
Ignore this finding from marshal-usage.
| @property | ||
| def meta(self) -> Dict[str, Any]: | ||
| """Get meta information (can not be changed after construction).""" | ||
| return {} if self._meta is None else cast(Dict[str, Any], loads(self._meta)) |
There was a problem hiding this comment.
The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. See more details: https://docs.python.org/3/library/marshal.html?highlight=security
Ignore this finding from marshal-usage.
| style._set_attributes = 0 | ||
| style._attributes = 0 | ||
| style._link = None | ||
| style._meta = dumps(meta) |
There was a problem hiding this comment.
The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. See more details: https://docs.python.org/3/library/marshal.html?highlight=security
Ignore this finding from marshal-usage.
| ) | ||
|
|
||
| self._link = link | ||
| self._meta = None if meta is None else dumps(meta) |
There was a problem hiding this comment.
The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source. See more details: https://docs.python.org/3/library/marshal.html?highlight=security
Ignore this finding from marshal-usage.
|
|
||
| context = SSLContext(ssl_version) | ||
|
|
||
| context.set_ciphers(ciphers or DEFAULT_CIPHERS) |
There was a problem hiding this comment.
The 'ssl' module disables insecure cipher suites by default. Therefore, use of 'set_ciphers()' should only be used when you have very specialized requirements. Otherwise, you risk lowering the security of the SSL channel.
Ignore this finding from no-set-ciphers.
| return self._ctx.set_npn_protocols(npn_protocols) | ||
|
|
||
| def set_ciphers(self, __cipherlist: str) -> None: | ||
| return self._ctx.set_ciphers(__cipherlist) |
There was a problem hiding this comment.
The 'ssl' module disables insecure cipher suites by default. Therefore, use of 'set_ciphers()' should only be used when you have very specialized requirements. Otherwise, you risk lowering the security of the SSL channel.
Ignore this finding from no-set-ciphers.
| self._connection = host, httplib.HTTPSConnection( | ||
| h, None, **kwargs) |
There was a problem hiding this comment.
The HTTPSConnection API has changed frequently with minor releases of Python. Ensure you are using the API for your version of Python securely. For example, Python 3 versions prior to 3.4.3 will not verify SSL certificates by default. See https://docs.python.org/3/library/http.client.html#http.client.HTTPSConnection for more information.
Ignore this finding from httpsconnection-detected.
| obj = obj.split(';') | ||
| # multi-line prep for generating an instance | ||
| for line in obj[:-1]: | ||
| exec(line) |
There was a problem hiding this comment.
Detected the use of exec(). exec() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from exec-detected.
| # This empty dict will contain the namespace for the exec'd file | ||
| custom_namespace = {} | ||
| with open(filename, 'rb') as f: | ||
| exec(f.read(), custom_namespace) |
There was a problem hiding this comment.
Detected the use of exec(). exec() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from exec-detected.
| # This empty dict will contain the namespace for the exec'd file | ||
| custom_namespace = {} | ||
| with open(filename, 'rb') as f: | ||
| exec(f.read(), custom_namespace) |
There was a problem hiding this comment.
Detected the use of exec(). exec() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from exec-detected.
| script_filename, | ||
| ) | ||
| script_code = compile(script_text, script_filename, 'exec') | ||
| exec(script_code, namespace, namespace) |
There was a problem hiding this comment.
Detected the use of exec(). exec() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from exec-detected.
| if os.path.exists(script_filename): | ||
| source = _read_utf8_with_fallback(script_filename) | ||
| code = compile(source, script_filename, 'exec') | ||
| exec(code, namespace, namespace) |
There was a problem hiding this comment.
Detected the use of exec(). exec() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from exec-detected.
| for line in obj[:-1]: | ||
| exec(line) | ||
| # one-line generation of an instance | ||
| obj = eval(obj[-1]) |
There was a problem hiding this comment.
Detected the use of eval(). eval() can be dangerous if used to evaluate dynamic content. If this content can be input from outside the program, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.
Ignore this finding from eval-detected.
| def clean(url): | ||
| "Tidy up an URL." | ||
| scheme, netloc, path, params, query, frag = urlparse(url) | ||
| return urlunparse((scheme, netloc, quote(path), | ||
| params, query, frag)) |
There was a problem hiding this comment.
function clean is defined inside a function but never used
| if problem: | ||
| deprecated( | ||
| reason=( | ||
| "Constraints are only allowed to take the form of a package " | ||
| "name and a version specifier. Other forms were originally " | ||
| "permitted as an accident of the implementation, but were " | ||
| "undocumented. The new implementation of the resolver no " | ||
| "longer supports these forms." | ||
| ), | ||
| replacement="replacing the constraint with a requirement", | ||
| # No plan yet for when the new resolver becomes default | ||
| gone_in=None, | ||
| issue=8210, | ||
| ) |
There was a problem hiding this comment.
Using strings as booleans in Python has unexpected results. "one" and "two" will return "two". "one" or "two" will return "one". In Python, strings are truthy, and strings with a non-zero length evaluate to True.
| host_ip = None | ||
| except AttributeError: | ||
| # Divergence from upstream: Make ipaddress library optional | ||
| if ipaddress is None: |
There was a problem hiding this comment.
Found identical comparison using is. Ensure this is what you intended.
Ignore this finding from identical-is-comparison.
| "implementation": implementation_info, | ||
| "system_ssl": system_ssl_info, | ||
| "using_pyopenssl": pyopenssl is not None, | ||
| "using_charset_normalizer": chardet is None, |
There was a problem hiding this comment.
Found identical comparison using is. Ensure this is what you intended.
Ignore this finding from identical-is-comparison.
| from urllib.error import HTTPError, URLError, ContentTooShortError | ||
| import http.client as httplib | ||
| import urllib.request as urllib2 | ||
| import xmlrpc.client as xmlrpclib |
There was a problem hiding this comment.
Detected use of xmlrpc. xmlrpc is not inherently safe from vulnerabilities. Use defusedxml.xmlrpc instead.
Ignore this finding from use-defused-xmlrpc.
| if ssl: | ||
| from urllib2 import HTTPSHandler | ||
| import httplib | ||
| import xmlrpclib |
There was a problem hiding this comment.
Detected use of xmlrpc. xmlrpc is not inherently safe from vulnerabilities. Use defusedxml.xmlrpc instead.
Ignore this finding from use-defused-xmlrpc.
| from pip._internal.network.utils import raise_for_status | ||
|
|
||
| if TYPE_CHECKING: | ||
| from xmlrpc.client import _HostType, _Marshallable |
There was a problem hiding this comment.
Detected use of xmlrpc. xmlrpc is not inherently safe from vulnerabilities. Use defusedxml.xmlrpc instead.
Ignore this finding from use-defused-xmlrpc.
|
|
||
| import logging | ||
| import urllib.parse | ||
| import xmlrpc.client |
There was a problem hiding this comment.
Detected use of xmlrpc. xmlrpc is not inherently safe from vulnerabilities. Use defusedxml.xmlrpc instead.
Ignore this finding from use-defused-xmlrpc.
| import shutil | ||
| import sys | ||
| import textwrap | ||
| import xmlrpc.client |
There was a problem hiding this comment.
Detected use of xmlrpc. xmlrpc is not inherently safe from vulnerabilities. Use defusedxml.xmlrpc instead.
Ignore this finding from use-defused-xmlrpc.
| ) | ||
|
|
||
| try: | ||
| subprocess.check_call(f'{editor} "{fname}"', shell=True) |
There was a problem hiding this comment.
Found 'subprocess' function 'check_call' with 'shell=True'. This is dangerous because this call will spawn the command using a shell process. Doing so propagates current shell settings and variables, which makes it much easier for a malicious actor to execute commands. Use 'shell=False' instead.
Ignore this finding from subprocess-shell-true.
| #name = os.path.splitext(os.path.basename(__file__))[0] | ||
| cProfile.run("dill.dumps(obj)", filename="%s.prof" % name) | ||
| msg = "gprof2dot -f pstats %s %s.prof | dot -Tpng -o %s.call.png" % (args, name, name) | ||
| os.system(msg) |
There was a problem hiding this comment.
Found user-controlled data used in a system call. This could allow a malicious actor to execute commands. Use the 'subprocess' module instead, which is easier to use without accidentally exposing a command injection vulnerability.
View Dataflow Graph
flowchart LR
classDef invis fill:white, stroke: none
classDef default fill:#e7f5ff, color:#1c7fd6, stroke: none
subgraph File0["<b>venv/bin/get_gprof</b>"]
direction LR
%% Source
subgraph Source
direction LR
v0["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L35 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 35] sys.argv</a>"]
end
%% Intermediate
subgraph Traces0[Traces]
direction TB
v2["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L35 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 35] obj</a>"]
v3["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L44 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 44] objtype</a>"]
v4["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L45 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 45] name</a>"]
v5["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L53 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 53] msg</a>"]
end
v2 --> v3
v3 --> v4
v4 --> v5
%% Sink
subgraph Sink
direction LR
v1["<a href=https://github.com/freelawproject/courtlistener/blob/838a19c03a5a138308c0001b497c48a443b2b5bc/venv/bin/get_gprof#L54 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 54] os.system(msg)</a>"]
end
end
%% Class Assignment
Source:::invis
Sink:::invis
Traces0:::invis
File0:::invis
%% Connections
Source --> Traces0
Traces0 --> Sink
| the backend might import. | ||
| """ | ||
|
|
||
| import importlib.resources as resources |
There was a problem hiding this comment.
Found 'importlib.resources', which is a module only available on Python 3.7+. This does not work in lower versions, and therefore is not backwards compatible. Use importlib_resources instead for older Python versions.
Ignore this finding from python37-compatibility-importlib2.
|
|
||
| elif sys.version_info >= (3, 7): | ||
|
|
||
| from importlib.resources import path as get_path, read_text |
There was a problem hiding this comment.
Found 'importlib.resources', which is a module only available on Python 3.7+. This does not work in lower versions, and therefore is not backwards compatible. Use importlib_resources instead for older Python versions.
Ignore this finding from python37-compatibility-importlib2.
| return add + text + sub | ||
| return "<" + add + ">" + text + "</" + sub + ">" |
There was a problem hiding this comment.
code after return statement will not be executed
Ignore this finding from code-after-unconditional-return.
| elif sys.version_info >= (3, 7): | ||
|
|
||
| from importlib.resources import path as get_path | ||
| from importlib.resources import read_text |
There was a problem hiding this comment.
Found 'importlib.resources', which is a module only available on Python 3.7+. This does not work in lower versions, and therefore is not backwards compatible. Use importlib_resources instead for older Python versions.
Ignore this finding from python37-compatibility-importlib2.
|
Semgrep found 6
Missing 'encoding' parameter. 'open()' uses device locale encodings by default, corrupting files with special characters. Specify the encoding to ensure cross-platform support when opening files in text mode (e.g. encoding="utf-8"). Ignore this finding from unspecified-open-encoding. |
i 've made a change in search/models, views, ran migrations, but somehow it generated changes to 558(!) files