HTMLGrabr library
A Node.js library to grab and clean HTML content.
Features
- Extract page content from an URL (
HTMLGrabr.grabURL(url: URL)) - Extract page content from a string (
HTMLGrabr.grab(s: string)) - Clean the page content:
- Extract main content using
node-readability - Extract text content using
html2plaintext - Remove link or image references of blacklisted sites
- Remove pixel tracker
- Remove unwanted attributes (by default:
classandid)
- Extract main content using
- Extract Open Graph properties
Usage
npm install --save htmlgrabrThe in your code:
const HTMLGrabr = require('htmlgrabr')
const { URL } = require('url')
const grabber = new HTMLGrabr()
grabber.grabUrl(new URL('http://keeper.nunux.org'))
.then(page => {
console.log(page)
}, err => {
console.log(err)
})Index
Classes
Interfaces
Variables
Functions
Object literals
Variables
Const DefaultFilterChain
Const db
Const h2p
Const pretty
Const readability
Functions
clean
-
Parameters
-
doc: Document
DOM to clean
-
props: CleanupProps
properties used by filters
-
Default value filters: FilterFunc[] = DefaultFilterChain
chain filter
Returns void
-
externalizeLinks
-
Update link to target new blank window.
Returns FilterFunc
the filtering function
extractBaseUrl
-
Extract base URL from headers of a DOM.
Parameters
-
doc: Document
DOM to process
Returns string | null
the base URL
-
extractImages
-
Extract all images from a DOM.
Parameters
-
doc: Document
DOM to process
-
Optional illustration: undefined | string
if proviided, the illustration is added to the result
Returns ImageMeta[]
array of image meta data
-
extractOpenGraphProps
-
Extract Open Graph properties from headers of a DOM.
Parameters
-
doc: Document
DOM to process
Returns OpenGraphProps
Open Graph properties
-
isBlacklisted
-
Parameters
-
hostname: string
Returns boolean
-
Const isValidUrl
-
Parameters
-
url: string
Returns boolean
-
moveAttribute
-
Move attribute to another.
Parameters
-
attr1: string
source attribute to remove
-
attr2: string
target attribute
Returns FilterFunc
the filtering function
-
rebaseSrcAttribute
-
Update relative link to be absolute.
Parameters
-
baseURL: string
the base URL used to make the link absolute
Returns FilterFunc
the filtering function
-
removeAttributes
-
Remove all blacklisted attributes from a HTML element.
Parameters
-
blacklist: string[]
the list of attributes to remove
Returns FilterFunc
the filtering function
-
removeBlacklistedLinks
-
Delete all HTML elements that refer to a blacklisted site.
Parameters
-
isBlacklisted: BlacklistCtrlFunc
Returns FilterFunc
the filtering function
-
removeImageTraker
-
Remove a image that can be a tracker (1px square).
Returns FilterFunc
the filtering function
Object literals
Const DefaultConfig
debug
headers
isBacklisted
Generated using TypeDoc
Clean a DOM using a filter chain.