Using node.io for screen scraping

I built a node.js prototype, which used node.io, for an idea we had to get some useful data.I knocked this out Sunday night and early Monday.
Besides learning node.io, a challenge was how to handle nested getHtml() calls in node.io since each is threaded. After 2 initial versions I found a useful post and the approach to use.

node.io is a powerful node.js package to support data manipulations (file io, db access, screen scraping etc.) in an async manner.
It has strong support for screen scraping which I used it for, I like the jquery like dom selector support, very cool. But I would definitely use it for file processing too.
In node.io you create jobs which are then run. You can define inputs and outputs. Lots of good material on the wiki.

The multi threading makes it a bit tricky especially for nested tasks.

Here's the git repo for the app I built.


A nod to my muse for this work, the "jean genie" maestro himself.


Comments

Post a Comment

Popular posts from this blog

deep dive into Material UI TextField built by mui

Image
The Material Design system is a design system used to create consistent digital experiences across platforms such as android, web and ios. Includes: a color system a typography system a shape system a grid system it addresses motion and lots more Also includes Material Theming which makes it easy to customize Material Design to match your brand. Material Design is not just a set of guidelines it is a whole ecosystem. Material itself stopped maintaining it's react implementation and recommends opensource versions.  A very popular open source implementation is  MUI ,  a react component library  built to implement Material Design . Bills itself as the "worlds most popular React UI framework" and at this time has 1.2M weekly downloads on npm and used by over 750k projects on github (did I mention it's very popular?).  Note: there are other react component libraries implementing bootstrap and ant design for example, so up to you and your design team on what you choose. 
Read more

angular js protractor e2e cheatsheet

Many from the protractor test suite category detailed task syntax example load page load a page browser.get('yoururl') browser btn actions browser back/forward browser.navigate().back() OR browser.navigate().forward() access elements on page See Protractor API list and examples from Protractors test suite access by angular ngmodel name (must match ngmodel in the html) this is the preferred approach element(by.model('person.name')); access by the css id on the page e.g, #user_name (but this can be brittle) element(by.id('user_name')) access angular binding to get generated value i.e. for {{}} element(by.binding('person.concatName')); access angular repeat list data (must match repeat stmt in the html) See Protractor test suit l
Read more

react-router v6.4+ loaders, actions, forms and more

react-router  react-router is used for in app single page app navigation, we configure routes for pages in the app to map to components react-router has over 1bn downloads, its an industry standard (but there are others e.g. Nextjs router) many apps use react-router, many on v5.x the new react-router 6.4.x the newest major version of react router v6.4 is a significant enhancement which adds a lot of new features. react-router v6.4 offers new features to load data (loaders) and handle actions (actions).  It's done gone all Remix-y which makes sense because the same guys wrote both. eliminates a lot of boilerplate code to fetch data in useEffect() and set various loading states put your data load and update logic in react-route components instead of custom code unlike Remix, react-router v6.4 is still just all on the client (not server like Remix) paradigm shifts Lets start with a couple of high level ideas; "back to the future" so to speak OR "what's old is new ag
Read more