Login to https://www.wuxiaworld.com

[dipu-bd]

You can login to WuxiaWorld now using the Bearer token. However it will be bit tricky to obtain the token for the beginners. I will describe the entire process here.

Step 1

Login to wuxiaworld using your browser (chrome is used for this example).

Step 2

Go to your profile page: https://www.wuxiaworld.com/manage/profile/

Step 3

Now open the "Developer Tools".

Step 4

Go to "Network" tab.

Step 5

Refresh the page

Step 6

Search for "GetMyUser" on "Network" tab filter option.

Step 7

Copy the value of the "authorization" header from the "Request Headers" section.

Step 8

Now you can access locked contents accessible to your wuxiaworld account using this oken. Open the lightnovel crawler application in your terminal and pass the token to the --login option. Here, "Bearer" is treated as user/email and the token following it is treated as password to the login option.

$ lncrawl --login Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjZBOEYzNTkwRDU2MDAyRDAxRDgzQUMyOTJFQjkyN0FENTk2NzFFMDBSUzI1NiIsIng1dCI6ImFvODFrTlZnQXRBZGc2d3BMcmtuclZsbkhnQSIsInR5cCI6ImF0K2p3dCJ9.eyJpc3MiOiJodHRwczovL2lkZW50aXR5Lnd1eGlhd29ybGQuY29tIiwibmJmIjoxNjUyNTY1MDgzLCJpYXQiOjE2NTI1NjUwODMsImV4cCI6MTY1NTE1NzA4MywiYXVkIjoiYXBpIiwic2NvcGUiOlsib3BlbmlkIiwicHJvZmlsZSIsImFwaSIsImVtYWlsIiwib2ZmbGluZV9hY2Nlc3MiXSwiYW1yIjpbInB3ZCJdLCJjbGllbnRfaWQiOiJ3dXhpYXdvcmxkX3NwYSIsInN1YiI6ImZlZjAzNWY5LWExZmUtNGU0ZC1hN2UwLTNjZmQwOWY4ZTgyMyIsImF1dGhfdGltZSI6MTY1MjU2NTA4MSwiaWRwIjoibG9jYWwiLCJuYW1lIjoidGlwb2ZtaW5kIiwiZW1haWwiOiJkaXB1LnN1ZGlwdGFAZ21haWwuY29tIiwic2lkIjoiM0JBMTlDQUFDQTUyRDJGNjUyNTY1MDRCMjc1QjZDQzYiLCJqdGkiOiJBNzk0RTNCMUVCRDMyNkZERTA2NDI1RjhEMEUzQTVBRCJ9.wUm9ot3wRopnVf64GNlZ7p-I8yfI7_fvEmuqfGgpbpVhWtXVOh9jSIErUugMn-0ofPMKKAi3LYtKBzpU2X5ZHH2xru-MgHqzBFadxJM2C3d9GfgPwwQTfvNv0m-Gj_7g9Vi0NY7Ck2kyjys8RJVmU2HE9_96VG8TGu9aB_qLBWVmh7N6szfmFbbjmad7OjWShb8RqdLFTL6AhhIWuZuuJoRzkmZzS-erkQNoxAIqXWS1Rj5Ji1Uc9JXbvig0DUytCkToilK3zptYCuAtRejjzvc6Ohs24Qfe3YXe3Tu-7xjM9vj_8xzZ2X9hlHlwfACad2YJ5XQN_5PLox4YcmzT-Xa95sTw-c9DvTpYKWBam0occsqB0JWyfLUJNJskRysA19H1T9HvOg2-GYpOFcM0iQ3TeOo9eY2QOuj75lKoxJJUKYlgsr9or-iK-dEpiJ7lodMCykICtVC5TzwK3Q17I1qVaxmd27wwbx2qaCcMOsJZeDcpxO09_Nx1zu2AnK1g-D8J5pNUey3tMZtS1cBRG_7S_T0qSwJY0rzuRTlaaE2IlKoAy0pJ4eoUp7mkL1OqMdw_RI-25SIUaTRIWxE3IgHOQTmwIxNk664oVJWMbxB9O38h7PB8egO8HzGbGIG1EM3kHfNoOSUDu1Ujc94wdHFD_9KwjpqTbN2K6biY8cI

[budikesuma]

@dipu-bd

Thank you for providing the login feature to wuxiaworld, I can login and download the chapters that have been unlocked first.
The only drawback is that it can't automatically unlock chapters that haven't been unlocked, even though I've set my account to unlock chapters automatically in the WW user settings.
Can the login feature still be developed?
I really appreciate it...

btw, I need to inform that I am successfully logged in on Termux, pyDroid3 and Google Colab too, either manually(by entering username&password) or with the --login Bearer xxxxxx --suppress command.
👍👍

[dipu-bd]

Enabling auto unlock will not be hard. But if I enable it, it will reduce karma points automatically. Some people may start complaining that they did not know it would make them lose karma points. I do not want to handle such hassle.

You can get VIP subscription to enable auto unlocks.

[ShakeBake]

Am I doing something wrong? I log in with the bearer key and point it to the novel, but the script opens a new chrome window which isn't logged in and just takes the teasers/cut-off chapters. I would log in to the new window but it disappears and creates a new one to start getting chapters.

[dipu-bd]

Browser log in is not supported in wuxiaworld

[vqidd]

@dipu-bd then why does it tell you if you want to log in when inserting a wuxiaworld link? Also the guest limit is also a thing.

[alzamer2]

@starxeras i know its late
but bearer log in is only supported in scrapper downloading which is blocked at moment because there is no Cloudflare by-pass method working

[alzamer2]

hi
i found this script in https://stackoverflow.com/questions/75723289/how-i-can-modify-header-in-seleniumwire-driver

from seleniumwire import webdriver  # Import from seleniumwire
import time
# Create a new instance of the Chrome driver (or Firefox)
driver = webdriver.Chrome()

# Create a request interceptor
def interceptor(request):
    #del request.headers['Referer']  # Delete the header first
    del request.headers['sec-ch-ua-mobile']  # Delete the header firs
    request.headers['sec-ch-ua-mobile'] = '?1'

# Set the interceptor on the driver
driver.request_interceptor = interceptor
# All requests will now use 'some_sec' for the user-agent
driver.get('https://httpbin.org/anything')
time.sleep(100)

cant we use it so we can login with browser in lightnovel-crawler?

[alzamer2]

hi @dipu-bd ,@ShakeBake
i updated wuxiacom.py so it can now login in browser
i need some testing/feedback befoer i do pull request

wuxiacom.zip

ps1: not sure why but wait timeout is not functioning as it should be, that why it take long time at locked chapter

ps2: is there a way to pass extra arg to sources so it can support extra functions, like auto-unlock chapters for example

ps3: fixed champion tab problem

[alzamer2]

@budikesuma i think this update support autologin in the WW user settings.

[krebsf01]

How does this work?

[alzamer2]

hi sorry for delay

first you need to go lncrawl source folder
its usually [python dir]\Lib\site-packages\sources\en\w
if you find wuxiacom.py replace it with attached file
but some time its on other directory so if you did not find it use "lncrawl -lll"
it will load debug mode
then add any novel url like "https://www.wuxiaworld.com/novel/nine-star-hegemon"
then it will show you where wuxiacom.py

[INFO] (lncrawl.core.sources)
Initializing crawler for: https://www.wuxiaworld.com/ [c:\users\****\appdata\local\programs\python\python311\Lib\site-packages\sources\en\w\wuxiacom.py]

"c:\users****\appdata\local\programs\python\python311\Lib\site-packages\sources\en\w\wuxiacom.py" is your wuxiacom.py directory
go into wuxiacom.py directory and replace it with attached file

after that add --login [your account email] [your account password] to commandline
or just run lncrawl and add novel url you want and it will ask you if you want to log in chose yes and add your account information

now if you see login page and auto login before it load novel info page then every thing is ok
note: some time at novel info page the page will stuck at champion tap, if it took too long switch manually to chapter page

after that it will ask you which chapter to download and your saving option

now when you finish everything and it start with scraping it will open login page again and auto login then it will start scraping and if your account setting you enabled "Enable Auto Unlock" it will will unlock all chapter you have access to (in theory champion chapter included it you have them)

ps: in selecting chapter and you did not "Enable Auto Unlock" in setting dont select locked chapter as the script is configured to wait for "Time Until Free Chapter" window to disappear and if you select them the code will wait for them to unlocked aka will stop at that page

[ARX-7]

Following on from this, assuming I have auto unlock disabled. I'd be able to feed thr crawler a series of novels that would scrape all avaliable chapters, unlock whatever time to wait chapters are avaliable and then stop at the next page?

In short use this to trigger what free chapters exist, use any "unlock credits" and allow me to automate the unlocking of free chapters every 23 hrs?

[alzamer2]

lightnovel-crawler was updated and now support login using username and Bearer even while using Browser

[X-Xadro]

This also works on MTLNation.com only you need to search for 'me' and its not 'Bearer' as username but 'JWT'

[PratikParmar810]

Hey, can we use this method on webnovel.com???

[alzamer2]

last time i checked webnovel use normal cookies to store username/password
but the real the problem in webnovel is its encryption for its chapters
so at moment only free chapters are scrap-able

[PratikParmar810]

@alzamer2
In webnovel, I can't generate epub or any file format.
In the generated Meta.json file, the chapters' body is null.

Also What about the "could not detect version_main.therefore, we are assuming it is chrome 108 or higher" shown in the first image in the Terminal?

[alzamer2]

about epub you need to install Calibre
https://github.com/dipu-bd/lightnovel-crawler?tab=readme-ov-file#installation
and make sure the calibre.exe path in your system PATH
and about "could not detect version_main.therefore, we are assuming it is chrome 108 or higher" you can ignore it, its waring msg from "undetected-chromedriver" pkg that lightnovel-crawler use, and it does not effect its running
but i'm asking to make sure you have normal chrome?

[PratikParmar810]

about epub you need to install Calibre https://github.com/dipu-bd/lightnovel-crawler?tab=readme-ov-file#installation and make sure the calibre.exe path in your system PATH and about "could not detect version_main.therefore, we are assuming it is chrome 108 or higher" you can ignore it, its waring msg from "undetected-chromedriver" pkg that lightnovel-crawler use, and it does not effect its running but i'm asking to make sure you have normal chrome?

@alzamer2
Yes, I am using normal Chrome, could not detect the message that showed up after I manually entered the "_crfToken" in the source file after I installed it using pip install.

[alzamer2]

``so here the thing
i'm trying to update scraping code so can it auto unlock chapter but i'm stuck at protobuff
here is the code:
response = self.grpc.request( f"{self.api_url}/wuxiaworld.api.v2.Unlocks/UnlockItem", {"unlockMethod": "UnlockMethodKarma", "item": { #"novelSlug": "/novel/nine-star-hegemon", #"chapterSlug": "nshba-chapter-5165" 'novelId': 76, 'entityId': 206310 } }, headers={ "origin": self.home_url, "referer": "https://www.wuxiaworld.com/novel/nine-star-hegemon/nshba-chapter-5165", "authorization": self.bearer_token, }, )
and its give me this error:

pyease_grpc.rpc_trailer.RpcTrailer: [UNKNOWN] gRPC Error: This chapter is already unlocked

and no the chapter is still locked and need karma to unloack