My experience using GenAI to scrape AirBnB

6

October

2024

No ratings yet.

Where I significantly noticed the productivity increase by the usage of Generative AI tools has been in my coding processes. During my bachelor degree, which was prior to the general launch of such GenAI tools, I learned coding “the hard way” and had to spend a lot of time on syntax- and semantics-errors, while generally only being able to create code for simpler tasks (e.g. to create a simple API connection).

Now, with the launch of innovative GenAI tools, I am suddently able to change the way I approach coding. Using for example the Microsoft Copilot which is embedded in the program “Visual Studio Code”, I am able to solve much more complex problems with a lot less effort. Using text-to-code tools enables “low-level-coders” like me to be way more productive and focus on creating the correct output instead of having to deal with syntax issues (McKinsey, 2023).

However, while these tools have greatly enhanced efficiency, I believe they are not perfect yet. For instance, I once had to build a web scraper for AirBnB to gather 100 listings, including their descriptions and prices per night. Building the general web scraper was quite easy, even though I had little prior experience. Nevertheless, achieving the final 5-10% of the quality required took considerable manual effort and time. In this case, while the initial code took about 30 minutes, I struggled to identify the correct HTML containers with Copilot’s help and had to find them myself (which took me hours…). As a result, I still needed to understand the code generated by GenAI, interpret what it was doing, and make adjustments on my own to achieve a fully functioning script.

(A quick look at the AirBnB Website and at one of the HTML divs I was trying to extract information from)

(an extract of the code I wrote; here you can see some “div”-examples I had to extract)

This experience highlighted to me that while Generative AI can automate many aspects of coding, a solid understanding of programming and the problem domain is still essential. AI-generated code is often a great starting point but rarely flawless. As a developer, you still have to refine the code and make adjustments where necessary, which required both technical knowledge and problem-solving skills.

In terms of improvement, I believe that integrating more advanced web page parsing capabilities and real-time feedback on the relevance of selected elements could further optimize the experience with such GenAI tools. Despite its limitations, Copilot’s potential to increase productivity in coding tasks is undeniable, and I look forward to seeing how it evolves.

Sources:

Please rate this

Leave a Reply

Your email address will not be published. Required fields are marked *