API reference¶
parsel.csstranslator¶
-
class
parsel.csstranslator.
GenericTranslator
[source]¶ Bases:
parsel.csstranslator.TranslatorMixin
,cssselect.xpath.GenericTranslator
-
class
parsel.csstranslator.
HTMLTranslator
(xhtml: bool = False)[source]¶ Bases:
parsel.csstranslator.TranslatorMixin
,cssselect.xpath.HTMLTranslator
-
class
parsel.csstranslator.
TranslatorMixin
[source]¶ Bases:
object
This mixin adds support to CSS pseudo elements via dynamic dispatch.
Currently supported pseudo-elements are
::text
and::attr(ATTR_NAME)
.-
xpath_attr_functional_pseudo_element
(xpath, function)[source]¶ Support selecting attribute values using ::attr() pseudo-element
-
parsel.selector¶
XPath selectors based on lxml
-
class
parsel.selector.
SafeXMLParser
(*args, **kwargs)[source]¶ Bases:
lxml.etree.XMLParser
-
class
parsel.selector.
Selector
(text: Optional[str, None] = None, type: Optional[str, None] = None, namespaces: Optional[Mapping[str, str], None] = None, root: Optional[Any, None] = None, base_url: Optional[str, None] = None, _expr: Optional[str, None] = None, huge_tree: bool = True)[source]¶ Bases:
object
Selector
allows you to select parts of an XML or HTML text using CSS or XPath expressions and extract data from it.text
is a str` objecttype
defines the selector type, it can be"html"
,"xml"
orNone
(default). Iftype
isNone
, the selector defaults to"html"
.base_url
allows setting a URL for the document. This is needed when looking up external entities with relative paths. See the documentation forlxml.etree.fromstring()
for more information.huge_tree
controls the lxml/libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion. The argument isTrue
by default if the installed lxml version supports it, which disables the protection to allow parsing such documents. Set it toFalse
if you want to enable the protection. See this lxml FAQ entry for more information.-
attrib
¶ Return the attributes dictionary for underlying element.
-
css
(query: str) → parsel.selector.SelectorList[_SelectorType][source]¶ Apply the given CSS selector and return a
SelectorList
instance.query
is a string containing the CSS selector to apply.In the background, CSS queries are translated into XPath queries using cssselect library and run
.xpath()
method.
-
extract
() → str¶ Serialize and return the matched nodes in a single string. Percent encoded content is unquoted.
-
get
() → str[source]¶ Serialize and return the matched nodes in a single string. Percent encoded content is unquoted.
-
namespaces
¶
-
re
(regex: Union[str, Pattern[str]], replace_entities: bool = True) → List[str][source]¶ Apply the given regex and return a list of strings with the matches.
regex
can be either a compiled regular expression or a string which will be compiled to a regular expression usingre.compile(regex)
.By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
-
re_first
(regex: Union[str, Pattern[str]], default: Optional[str, None] = None, replace_entities: bool = True) → Optional[str, None][source]¶ Apply the given regex and return the first string which matches. If there is no match, return the default value (
None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
-
register_namespace
(prefix: str, uri: str) → None[source]¶ Register the given namespace to be used in this
Selector
. Without registering namespaces you can’t select or extract data from non-standard namespaces. See Working on XML (and namespaces).
-
remove_namespaces
() → None[source]¶ Remove all namespaces, allowing to traverse the document using namespace-less xpaths. See Removing namespaces.
-
root
¶
-
selectorlist_cls
¶ alias of
SelectorList
-
text
¶
-
type
¶
-
xpath
(query: str, namespaces: Optional[Mapping[str, str], None] = None, **kwargs) → parsel.selector.SelectorList[_SelectorType][source]¶ Find nodes matching the xpath
query
and return the result as aSelectorList
instance with all elements flattened. List elements implementSelector
interface too.query
is a string containing the XPATH query to apply.namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
-
-
class
parsel.selector.
SelectorList
[source]¶ Bases:
list
,typing.Generic
The
SelectorList
class is a subclass of the builtinlist
class, which provides a few additional methods.-
attrib
¶ Return the attributes dictionary for the first element. If the list is empty, return an empty dict.
-
css
(query: str) → parsel.selector.SelectorList[_SelectorType][source]¶ Call the
.css()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.css()
-
extract
() → List[str]¶ Call the
.get()
method for each element is this list and return their results flattened, as a list of strings.
-
extract_first
(default: Optional[str, None] = None) → Optional[str, None]¶ Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
-
get
(default: Optional[str, None] = None) → Optional[str, None][source]¶ Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
-
getall
() → List[str][source]¶ Call the
.get()
method for each element is this list and return their results flattened, as a list of strings.
-
re
(regex: Union[str, Pattern[str]], replace_entities: bool = True) → List[str][source]¶ Call the
.re()
method for each element in this list and return their results flattened, as a list of strings.By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
-
re_first
(regex: Union[str, Pattern[str]], default: Optional[str, None] = None, replace_entities: bool = True) → Optional[str, None][source]¶ Call the
.re()
method for the first element in this list and return the result in an string. If the list is empty or the regex doesn’t match anything, return the default value (None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
-
xpath
(xpath: str, namespaces: Optional[Mapping[str, str], None] = None, **kwargs) → parsel.selector.SelectorList[_SelectorType][source]¶ Call the
.xpath()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.xpath()
namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
-
parsel.utils¶
-
parsel.utils.
extract_regex
(regex: Union[str, Pattern[str]], text: str, replace_entities: bool = True) → List[str][source]¶ Extract a list of strings from the given text/encoding using the following policies: * if the regex contains a named group called “extract” that will be returned * if the regex contains multiple numbered groups, all those will be returned (flattened) * if the regex doesn’t contain any group the entire regex matching is returned
-
parsel.utils.
flatten
(sequence) → list[source]¶ Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables). Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten([“foo”, “bar”]) [‘foo’, ‘bar’] >>> flatten([“foo”, [“baz”, 42], “bar”]) [‘foo’, ‘baz’, 42, ‘bar’]