Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requirements.txt not able to be analyzed #7127

Open
h4ckerfr0st opened this issue Nov 1, 2024 · 2 comments
Open

Requirements.txt not able to be analyzed #7127

h4ckerfr0st opened this issue Nov 1, 2024 · 2 comments
Labels

Comments

@h4ckerfr0st
Copy link

h4ckerfr0st commented Nov 1, 2024

Issue analyzing requirements.txt

Basically i have a project in which i am running o gitlab ci/cd in Docker and im encountering an error when it comes to analyze requirements.txt that owasp_dependency_check informs.

What have i done until now based on the error below whilst running this command:

  • Remove any analogous character : cat -A /app/requirements.txt
  • Identify encoding now at this moment is : requirements.txt: text/plain; charset=us-ascii, same as utf-8 baiscally
  • run different places and commands
  • Run a different requirements.txt from other sources

This is my requirements.txt :

boto3==1.24.32
sqlalchemy==1.3.24
psycopg2==2.8.6
ujson==1.35
flask==1.0.3
flask-cors==3.0.9
pytz==2023.3.post1
cryptography==41.0.5
jira==2.0.0
requests==2.25.1
elasticsearch==7.14.0
elasticsearch-dsl==7.4.0
XlsxWriter==3.2.0
aiohttp==3.10.5
async-timeout==4.0.3
attrs==20.3.0
botocore==1.27.32
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==8.1.3
debtcollector==2.2.0
defusedxml==0.7.1
eight==1.0.1
expiringdict==1.1.4
fasteners==0.16
future==0.18.2
idna==2.9
importlib-metadata==4.3.0
IPy==1.01
iso8601==0.1.14
itsdangerous==1.1.0
Jinja2==2.11.3
jmespath==0.10.0
MarkupSafe==1.1.1
motor==2.4.0
multidict==4.7.6
netaddr==0.8.0
netifaces==0.10.9
oauthlib==3.1.0
oslo.concurrency==3.25.0
oslo.config==8.5.0
oslo.i18n==5.0.1
oslo.utils==4.8.0
packaging==20.9
pbr==5.5.1
pycparser==2.20
PyJWT==2.0.1
pymongo==3.13.0
pyparsing==2.4.7
python-dateutil==2.7.3
PyYAML==6.0.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rfc3986==1.4.0
s3transfer==0.6.0
six==1.15.0
stevedore==3.3.0
tenacity==5.1.4
typing==3.5.3.0
typing-extensions==3.7.4.3
urllib3==1.26.9
uWSGI==2.0.22
watchtower==0.8.0
Werkzeug==1.0.1
wrapt==1.14.1
yarl==1.6.3
zipp==3.11.0
anyio==3.5.0
charset-normalizer==3.3.2
h11==0.14.0
httpcore==0.17.0
httpx==0.24.0
sniffio==1.2.0
retry==0.9.2
decorator==5.1.1
elastic-apm[flask]>=6.8.1,<7
blinker>=1.5,<1.6

Below is my gitlab CI/CD pipeline.

dependency_check:
image: xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/appsec/python_analyzer:latest
stage: security
script:
- tr -d '$' < requirements.txt > requirements_cleaned.txt
- mv requirements_cleaned.txt requirements.txt
- dependency-check --disableAssembly --enableExperimental -s . -o build --format XML --project "xxxxxxxx" --exclude ".git/" --exclude ".venv/" --exclude "/pycache/" --exclude ".tox/**" --nvdApiKey "xxxxxxxxxxxxx"

About ODC: https://jeremylong.github.io/DependencyCheck/general/internals.html
False Positives: https://jeremylong.github.io/DependencyCheck/general/suppression.html
? Sponsor: https://github.com/sponsors/jeremylong
[INFO] Analysis Started
[INFO] Finished Archive Analyzer (0 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Python Distribution Analyzer (0 seconds)
[INFO] Finished Python Package Analyzer (0 seconds)
[WARN] An unexpected error occurred during analysis of '/app/requirements.txt' (pip Analyzer): java.nio.charset.MalformedInputException: Input length = 1
[ERROR]
java.io.UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1
at java.base/java.nio.file.FileChannelLinesSpliterator.readLine(FileChannelLinesSpliterator.java:176)
at java.base/java.nio.file.FileChannelLinesSpliterator.forEachRemaining(FileChannelLinesSpliterator.java:116)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.owasp.dependencycheck.analyzer.PipAnalyzer.getFileContents(PipAnalyzer.java:188)
at org.owasp.dependencycheck.analyzer.PipAnalyzer.analyzeDependency(PipAnalyzer.java:140)
at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze(AbstractAnalyzer.java:131)
at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:88)
at org.owasp.dependencycheck.AnalysisTask.call(AnalysisTask.java:37)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:274)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:350)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:179)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
at java.base/java.nio.file.FileChannelLinesSpliterator.readLine(FileChannelLinesSpliterator.java:174)

Thanks for the help in the mean time if possible, please.

@h4ckerfr0st h4ckerfr0st added the bug label Nov 1, 2024
@aikebah aikebah added question and removed bug labels Nov 2, 2024
@aikebah
Copy link
Collaborator

aikebah commented Nov 2, 2024

Based on your symptoms my gut feel would be that your docker container is configured with a multi-byte characterset as the platform default encoding.
The requirements.txt is read using the 'default charset' as established by the Java Charset class.

return Files.lines(actualFile.toPath(), Charset.defaultCharset())

@aikebah aikebah added bug and removed question labels Nov 2, 2024
@rdimameza
Copy link

rdimameza commented Nov 12, 2024

I had the same issue and I could fix it adding the next two steps:

  • name: Set default charset to UTF-8
    run: |
    export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8"

  • name: Convert requirements.txt to UTF-8
    run: |
    iconv -f -t UTF-8 requirements.txt -o requirements.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants