Using BeautifulSoup to scrape lyrics might be painful

[AnveshakR]

Hey, first off, great concept, and I'm really flattered for the mention in your Readme. I was looking through your code to try and get some posters for myself. I wasn't able to get the lyrics for quite a few songs, like Money Trees by Kendrick Lamar. I was poking through the lyrics fetcher to see how you're going about it and I saw that you were using Beautifulsoup to get the website content for the lyrics.

Speaking from personal experience, using the class name to get the divs containing the lyrics are you are in cols = soup.findAll(class_="lyrics__content__ok", string=True) might not be the best idea as these websites often have computer generated class names, which means your website might break every time they decide to update the website. I know it because I faced the same issue with a Google Meet automator that I made a few years ago.

You could instead look into finding divs with words that always occur in the lyrics content, like "Verse", and try finding the parent class/div for the same. Just a few ideas for you to explore.

[TrueMyst]

I've faced this problem too, so I decided to make a project that somewhat fixes it, by not using a scraper in the first place, because that seems somewhat inefficient, it's not finished yet.

I'm planning to add other lyrics sources to it, because MusixMatch in general sucks. LyricsFind seems somewhat reliable for this project, and I feel like we can use it to fix this issue above.

For example, Money Trees by Kendrick Lamar does exists on LyricsFind, and I plan to add this source too.

To be fair, I'm really busy these days, so I don't get that much time to work on this project 😅. But I'm trying my best to work on these little. If you want, we can work together on this feature. That'll be great ✨!

[AnveshakR]

Oh, I wrote a fix just now out of sheer boredom. Do you want me to send a pull request?

[TrueMyst]

Yeah sure! That'll be really awesome mate :D

[AnveshakR]

submitted

[TrueMyst]

That's great! Thanks for submitting the PR, that was really helpful.

I was thinking if we could work on fetching the lyrics from other sources too, for example LyricsFind? That'll be great right? Because currently I'm working on pymusix.

MusixMatch is not the ideal way to fetch lyrics to be honest, because even if you submit the lyrics it'll take more than year to get them approved (I've tried many times).

I've tried working with Genius and it is really bad too.

Your PR was great and I really appreciate it but I was thinking if could move it out into a different project where fetching the lyrics would be much easier you know? Because I want to make this project more flexible and not break instead.

[AnveshakR]

well, i guess Musixmatch gives uploading priority to the artist over the general public. I haven't personally heard of LyricsFind, but if they have a better developer access then it should be much easier for sure. I use Spotify for my music and pretty much all of the songs I listen to have lyrics listed by Musixmatch, so I doubt there is much they don't cover. But of course, do look at other options; it is your project after all :)

[TrueMyst]

I use Spotify for my music and pretty much all of the songs I listen to have lyrics listed by Musixmatch, so I doubt there is much they don't cover

I agree, but not all music on Spotify have lyrics listed by MusixMatch which is a problem, for example Hindi music. Some of them don't even appear due to copyright issues. Check this out for example.

I don't listen to Hindi music but a lot of people were complaining about this issue.