Developing sanctions crawlers
If this is your first crawler, start with a basic one by following the tutorial, then come back here. Program keys are covered in programs, and short worked crawlers in examples.
How a designation is modeled
Each designation is the designated entity plus a Sanction built by
h.make_sanction. Add topics: sanction to the
designated entity, and emit it after the Sanction is made (see
patterns).
An entity designated under several programs gets one Sanction per program, told apart
by key= (see sub-entity IDs).
Assert Sanction counts alongside the Person and Organization counts (see
data assertions).
Sanction properties
h.make_sanction fills entity, authority, country
and sourceUrl from the dataset metadata. The crawler fills the rest from the source:
authorityId: the issuing authority's own reference for the designation.programandprogramId: see programs.provisions: the restrictions imposed by the designation.reason: the source's statement of grounds.
Dates
The date properties describe the designation as the authority made it, not our processing of it:
listingDate: when the designation was listed.startDate: when it took effect.endDate: when it expires, usually in the future.modifiedAt: when the authority last amended the designation, or when it was listed if it was never amended.
An entity's last_change is separate: it records when our output for the entity
last changed, including changes caused by our own parsing.
Fill each one the source gives. Before mapping a source date to modifiedAt, check
that the authority updates it when it amends the designation, and only then. Don't
set createdAt, and never use the time of the current crawl for any of these.
Call h.finalize_sanction after setting the
dates and before emitting the Sanction. When the legal instrument makes designations
on a list take effect when listed, pass effective_on_listing=True: it fills
whichever of listingDate and startDate is missing from the other. When the source
dates every amendment, pass amendments_dated=True: it fills a missing modifiedAt
from the listing. Check both per list.
When the source lists a designation whose endDate has passed, emit it, and set its
topics as described in topics.
Choosing the schema
Personfor individuals andVesselfor ships.LegalEntitywhen the source does not say whether the target is a person or an organization.Organizationfor associations and social constructs, such as Al Qaeda, listed among companies. Government departments areOrganizationtoo; an upcast toCompanyis acceptable.
See also reference entities that fit the property range.
Entity IDs
Use the source's own ID once you have checked that it is reasonably reliable. Otherwise,
pass context.make_id a sufficiently unique set of unparsed source values, such as the
name, date of birth and jurisdiction. See entity IDs.
Names
New sanctions crawlers add names through h.apply_reviewed_names. Base the
names.schema_rules in the dataset YAML on what you find when studying the source's
names. See names.
De-listings
Drop de-listed entities. Some policies make specific exceptions to this.
Linked parties
Emit linked parties, such as owners, parent companies or crypto wallets, without
topics: the analyzers compute them. Use UnknownLink only where the source asserts a
link without saying what kind it is.
Lists that republish UN or EU designations
Reuse the shared parsers in zavod.shed:
zavod.shed.un_sc:load_un_scfetches the UN Security Council list, andget_personsandget_legal_entitiesfilter it byRegime, keying entities with theun_sc_sanctionsprefix so they share IDs with that dataset.apply_un_name_listadds UN-style name parts.zavod.shed.fsf:parse_entryparses an entry of the EU Financial Sanctions Files XML, afterh.remove_namespaceon the document.
Lists without a structured source
See curating a dataset.