life is short - you need Python!

I just wrote a small Python program. In the script there was a part where I needed to get the body of a web page and get rid of all the html tags, javascript, css styles, html comments etc. So I searched Google, found several threads in stackoverflow and then found this: http://www.aaronsw.com/2002/html2text/ This looks cool. But when I tested it against the 'about me' page of my blog, it didn't work because of some broken tags! Then I started to write the html to text function myself to get the plain text only. With help of regular expression I solved my problem (but may be I created more problems!). Here is my Python code: def html_to_text(data):              # remove the newlines     data = data.replace("\n", " ")     data = data.replace("\r", " ")        # replace consecutive spaces into a single one     data = " ".join(data.split())    ...

'How to learn regular expression in Python?' - a very common question from the beginners or Python newbies. Well... if you can use regular expression in Perl or PHP, then it's not very different in Python. And if you are completely new to regular expression and want to learn it, then you can follow the following guideline: [Update on Oct 30, 2010]: I think you should start with this video on regular expression . At first, you should be to read Regular Expression HowTo by A.M. Kuchling. Read it carefully and read it at least twice :) So... now you have finished reading Regular Expression How To and still not very confident. Don't worry. It's time to dive into regular expression. Just go through Chapter 7 Regular Expressions of Dive Into Python. Don't forget to code each and every example. There are some nice case studies. Those will help (it was really helpful in my case). And now it's time for you to check details about the re module from python...

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags: def remove_html_tags(data): p = re.compile(r'<.*?>') return p.sub('', data) Here is another function to remove more than one consecutive white spaces: def remove_extra_spaces(data): p = re.compile(r'\s+') return p.sub(' ', data) Note that re module needs to be imported in order to use regular expression. Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html