The Job Seeker project and other odds and sods
Hey all, reporting back with news about our project: Ana, Brandon and I have been busy preparing the data pipeline from scraping to analysis: - scraping we search for a specific job title on LinkedIn (e.g., Data Engineer) and scrape X number of search results pages. we save the resulting data (e.g., title, location, description, etc.) into a json file with help of the pydantic module - skills extraction we prompt the LLM to extract required skills from the job descriptions. again with the help of pydantic, we save the resulting data into a json file. - skills clustering the resulting json is far too detailed to be useful, so we prompt the LLM to evaluate the skills and identify and name clusters of skills. pydantic, json, file. - data processing we have taken a first shot at processing this data: dimensionality reduction, association rule mining, network analysis. we are just at the beginning, but it's super inspiring that we got this far in our own project. it's too early to report findings, but once we find something interesting, we'll let you know :) I recommend everybody to find like-minded community members and embark on projects where you can contribute, learn and co-operate. It's motivating to know that you are on a joint endeavour. Speaking of which: thanks to Zachary, our team was able to reach out to an organisation in East Africa that helps farmers turn to organic farming. We had our first meeting yesterday and learnt so much about their objectives and the problems they face. We are brainstorming on what project we can design so we learn, collaborate and contribute. Ana is extremely talented in bringing people together and materialising opportunities. It's thanks to her enthusiasm that we got this far with the organic farming project! Over and out, back to enjoying ourselves on our self initiated projects :)