API reference
parsel.csstranslator
- class parsel.csstranslator.GenericTranslator[source]
Bases:
TranslatorMixin
,GenericTranslator
- css_to_xpath(css: str, prefix: str = 'descendant-or-self::') str [source]
Translate a group of selectors to XPath.
Pseudo-elements are not supported here since XPath only knows about “real” elements.
- Parameters:
css – A group of selectors as a string.
prefix – This string is prepended to the XPath expression for each selector. The default makes selectors scoped to the context node’s subtree.
- Raises:
SelectorSyntaxError
on invalid selectors,ExpressionError
on unknown/unsupported selectors, including pseudo-elements.- Returns:
The equivalent XPath 1.0 expression as a string.
- class parsel.csstranslator.HTMLTranslator(xhtml: bool = False)[source]
Bases:
TranslatorMixin
,HTMLTranslator
- css_to_xpath(css: str, prefix: str = 'descendant-or-self::') str [source]
Translate a group of selectors to XPath.
Pseudo-elements are not supported here since XPath only knows about “real” elements.
- Parameters:
css – A group of selectors as a string.
prefix – This string is prepended to the XPath expression for each selector. The default makes selectors scoped to the context node’s subtree.
- Raises:
SelectorSyntaxError
on invalid selectors,ExpressionError
on unknown/unsupported selectors, including pseudo-elements.- Returns:
The equivalent XPath 1.0 expression as a string.
- class parsel.csstranslator.TranslatorMixin[source]
Bases:
object
This mixin adds support to CSS pseudo elements via dynamic dispatch.
Currently supported pseudo-elements are
::text
and::attr(ATTR_NAME)
.- xpath_attr_functional_pseudo_element(xpath: XPathExpr, function: FunctionalPseudoElement) XPathExpr [source]
Support selecting attribute values using ::attr() pseudo-element
- xpath_pseudo_element(xpath: XPathExpr, pseudo_element: FunctionalPseudoElement | str) XPathExpr [source]
Dispatch method that transforms XPath to support pseudo-element
parsel.selector
XPath and JMESPath selectors based on the lxml and jmespath Python packages.
- class parsel.selector.Selector(text: str | None = None, type: str | None = None, body: bytes = b'', encoding: str = 'utf8', namespaces: ~typing.Mapping[str, str] | None = None, root: ~typing.Any | None = <object object>, base_url: str | None = None, _expr: str | None = None, huge_tree: bool = True)[source]
Bases:
object
Wrapper for input data in HTML, JSON, or XML format, that allows selecting parts of it using selection expressions.
You can write selection expressions in CSS or XPath for HTML and XML inputs, or in JMESPath for JSON inputs.
text
is anstr
object.body
is abytes
object. It can be used together with theencoding
argument instead of thetext
argument.type
defines the selector type. It can be"html"
(default),"json"
, or"xml"
.base_url
allows setting a URL for the document. This is needed when looking up external entities with relative paths. See the documentation forlxml.etree.fromstring()
for more information.huge_tree
controls the lxml/libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion. The argument isTrue
by default if the installed lxml version supports it, which disables the protection to allow parsing such documents. Set it toFalse
if you want to enable the protection. See this lxml FAQ entry for more information.- body
- css(query: str) SelectorList[_SelectorType] [source]
Apply the given CSS selector and return a
SelectorList
instance.query
is a string containing the CSS selector to apply.In the background, CSS queries are translated into XPath queries using cssselect library and run
.xpath()
method.
- extract() Any
Serialize and return the matched nodes.
For HTML and XML, the result is always a string, and percent-encoded content is unquoted.
- get() Any [source]
Serialize and return the matched nodes.
For HTML and XML, the result is always a string, and percent-encoded content is unquoted.
- jmespath(query: str, **kwargs: Any) SelectorList[_SelectorType] [source]
Find objects matching the JMESPath
query
and return the result as aSelectorList
instance with all elements flattened. List elements implementSelector
interface too.query
is a string containing the JMESPath query to apply.Any additional named arguments are passed to the underlying
jmespath.search
call, e.g.:selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
- namespaces
- re(regex: str | Pattern[str], replace_entities: bool = True) List[str] [source]
Apply the given regex and return a list of strings with the matches.
regex
can be either a compiled regular expression or a string which will be compiled to a regular expression usingre.compile(regex)
.By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
- re_first(regex: str | Pattern[str], default: None = None, replace_entities: bool = True) str | None [source]
- re_first(regex: str | Pattern[str], default: str, replace_entities: bool = True) str
Apply the given regex and return the first string which matches. If there is no match, return the default value (
None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
- register_namespace(prefix: str, uri: str) None [source]
Register the given namespace to be used in this
Selector
. Without registering namespaces you can’t select or extract data from non-standard namespaces. See Working on XML (and namespaces).
- remove_namespaces() None [source]
Remove all namespaces, allowing to traverse the document using namespace-less xpaths. See Removing namespaces.
- selectorlist_cls
alias of
SelectorList
[Selector]
- type
- xpath(query: str, namespaces: Mapping[str, str] | None = None, **kwargs: Any) SelectorList[_SelectorType] [source]
Find nodes matching the xpath
query
and return the result as aSelectorList
instance with all elements flattened. List elements implementSelector
interface too.query
is a string containing the XPATH query to apply.namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
- class parsel.selector.SelectorList(iterable=(), /)[source]
Bases:
List
[_SelectorType
]The
SelectorList
class is a subclass of the builtinlist
class, which provides a few additional methods.- property attrib: Mapping[str, str]
Return the attributes dictionary for the first element. If the list is empty, return an empty dict.
- css(query: str) SelectorList[_SelectorType] [source]
Call the
.css()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.css()
- extract() List[str]
Call the
.get()
method for each element is this list and return their results flattened, as a list of strings.
- extract_first(default: str | None = None) Any
Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
- get(default: None = None) str | None [source]
- get(default: str) str
Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
- getall() List[str] [source]
Call the
.get()
method for each element is this list and return their results flattened, as a list of strings.
- jmespath(query: str, **kwargs: Any) SelectorList[_SelectorType] [source]
Call the
.jmespath()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.jmespath()
.Any additional named arguments are passed to the underlying
jmespath.search
call, e.g.:selector.jmespath('author.name', options=jmespath.Options(dict_cls=collections.OrderedDict))
- re(regex: str | Pattern[str], replace_entities: bool = True) List[str] [source]
Call the
.re()
method for each element in this list and return their results flattened, as a list of strings.By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
- re_first(regex: str | Pattern[str], default: None = None, replace_entities: bool = True) str | None [source]
- re_first(regex: str | Pattern[str], default: str, replace_entities: bool = True) str
Call the
.re()
method for the first element in this list and return the result in an string. If the list is empty or the regex doesn’t match anything, return the default value (None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
- xpath(xpath: str, namespaces: Mapping[str, str] | None = None, **kwargs: Any) SelectorList[_SelectorType] [source]
Call the
.xpath()
method for each element in this list and return their results flattened as anotherSelectorList
.xpath
is the same argument as the one inSelector.xpath()
namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
parsel.utils
- parsel.utils.extract_regex(regex: str | Pattern[str], text: str, replace_entities: bool = True) List[str] [source]
Extract a list of strings from the given text/encoding using the following policies: * if the regex contains a named group called “extract” that will be returned * if the regex contains multiple numbered groups, all those will be returned (flattened) * if the regex doesn’t contain any group the entire regex matching is returned
- parsel.utils.flatten(sequence) list [source]
Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables). Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten([“foo”, “bar”]) [‘foo’, ‘bar’] >>> flatten([“foo”, [“baz”, 42], “bar”]) [‘foo’, ‘baz’, 42, ‘bar’]