-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pull request to add robots.txt and sitemap.xml files (v1) #643
base: main
Are you sure you want to change the base?
Conversation
User-agent: * | ||
Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password | ||
Disallow: /examples/hash_password.sql # Block access to specific sensitive SQL files like this | ||
Disallow: /Dockerfile # Block access to Dockerfile |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was told to limit the number of files that need to be crawled so I figure this one is not useful for Google
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Look at the link. The file does not exist
@@ -0,0 +1,8 @@ | |||
User-agent: * | |||
Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password | |||
Disallow: /examples/hash_password.sql # Block access to specific sensitive SQL files like this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why ? This is a useful page, it is not password protected: https://sql.datapage.app/examples/hash_password.sql
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which one is ?
@@ -0,0 +1,8 @@ | |||
User-agent: * | |||
Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe do not block the entire authentication example ? We still want it referenced. Ideally change the auth example so the password protected page has a different url, that you can block
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> | ||
<url> | ||
<loc>https://sql.datapage.app/</loc> | ||
<lastmod>2024-10-10</lastmod> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it ok to hardcode the date like this ? Won't the prevent re-crawling when we update the page ?
<loc>https://sql.datapage.app/your-first-sql-website/</loc> | ||
<lastmod>2024-10-10</lastmod> | ||
<priority>0.80</priority> | ||
</url> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it ok to have such a little subset of the site in the sitemap ? Won't it penalize other pages ?
We should probably either not have a sitemap, or have one that is generated dynamically in sql
No description provided.