Skip to content

Name cleaning

One or more names are generally available for entities named/listed in our data sources. We often need to

  • categorise them, e.g. as primary name, aliases, or previous/former names,
  • clean superfluous text that is not part of the name.

Clean, correctly categorised names are important to maximise recall (finding all true matches) and maximise precision (avoiding false positives).

While we've done this using simple, explainable logic for the most part, this leaves some noise or incorrectly-categorised names for a number of sources.

What's a dirty name?

The helper zavod.helpers.is_name_irregular returns true if a name potentially needs cleaning.

A dataset can customise what should be considered "in need of cleaning" using options under the names key of the dataset metadata. Each field under names is a schema type, so that different rules can apply to different entities in the dataset.

e.g.

names:
  Company:
    reject_chars: ","
    allow_chars: "/"

zavod.meta.names.NamesSpec

Name cleaning requirements by schema. All matching schema configurations will apply

Source code in zavod/meta/names.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
class NamesSpec(RootModel[Dict[str, CleaningSpec]]):
    """Name cleaning requirements by schema. All matching schema configurations will apply"""

    ###################
    # Beware that when introducing defaults for more specific schemata, these could take
    # precedence over extensions to the existing defaults in some datasets' metadata.
    # Those datasets might need to be updated to extend the new default instead.
    root: Dict[str, CleaningSpec] = {
        "Person": CleaningSpec(
            reject_chars_baseline=";\\/()[]<>{}:",
            require_space=True,
        ),
        "LegalEntity": CleaningSpec(
            reject_chars_baseline="/;",
        ),
        "Vessel": CleaningSpec(
            reject_chars_baseline="/;",
        ),
    }

    @classmethod
    def model_validate(cls, obj: Any, **kwargs: Any) -> "NamesSpec":
        """Merge provided values with defaults."""
        if isinstance(obj, dict):
            instance = cls()
            for schema_name, spec in obj.items():
                if schema_name in instance.root:
                    schema = Model.instance().get(schema_name)
                    assert schema is not None, schema_name
                    # Merge with default
                    default_spec = instance.root[schema_name]
                    merged_spec = default_spec.model_copy(update=spec)
                    instance.root[schema_name] = merged_spec
                else:
                    instance.root[schema_name] = CleaningSpec.model_validate(spec)
            return instance
        raise TypeError(f"object must be a dict, got {type(obj)}")

    def get_spec(self, schema: Schema) -> Optional[CleaningSpec]:
        """Returns the spec for the most specific schema that matches the entity."""
        matching_specs = [
            (Model.instance().get(name), spec)
            for name, spec in self.root.items()
            if schema.is_a(name)
        ]
        # schema names validated in model_validate
        specs = [(s, spec) for s, spec in matching_specs if s is not None]
        specs.sort(key=lambda pair: len(pair[0].schemata), reverse=True)
        # We don't support multiple inheritance for now. Unlikely to define a spec for Asset.
        return specs[0][1] if specs else None

zavod.meta.names.CleaningSpec

Source code in zavod/meta/names.py 
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
class CleaningSpec(BaseModel):
    reject_chars_baseline: str = ""
    """The standard characters that suggest a name needs cleaning."""
    reject_chars: str = ""
    """
    Additional characters specific to this schema that suggest a name needs cleaning.

    Use this to define characters in dataset-specific config. Adds to the baseline
    characters for default specs.
    """
    allow_chars: str = ""
    """
    Characters that would otherwise trigger cleaning but are allowed for this schema.

    Remember that characters defined for other matching schema specs will still apply.
    """
    min_chars: int = 2
    single_token_min_length: int = 2
    """Minimum length for names with no spaces, i.e. a single token."""
    require_space: bool = False
    allow_nullwords: bool = False

    @cached_property
    def reject_chars_consolidated(self) -> Set[str]:
        """Get the full set of characters to reject for this spec."""
        baseline = set(self.reject_chars_baseline)
        reject_extra = set(self.reject_chars)
        allow = set(self.allow_chars)
        return (baseline | reject_extra) - allow
allow_chars = '' class-attribute instance-attribute

Characters that would otherwise trigger cleaning but are allowed for this schema.

Remember that characters defined for other matching schema specs will still apply.

allow_nullwords = False class-attribute instance-attribute
min_chars = 2 class-attribute instance-attribute
reject_chars = '' class-attribute instance-attribute

Additional characters specific to this schema that suggest a name needs cleaning.

Use this to define characters in dataset-specific config. Adds to the baseline characters for default specs.

reject_chars_baseline = '' class-attribute instance-attribute

The standard characters that suggest a name needs cleaning.

reject_chars_consolidated cached property

Get the full set of characters to reject for this spec.

require_space = False class-attribute instance-attribute
single_token_min_length = 2 class-attribute instance-attribute

Minimum length for names with no spaces, i.e. a single token.

Using LLMs

LLMs can do a lot of the categorisation and cleaning for us. We pair this with human reviews to make 100% sure the categorisation and cleaning was correct, and did not lose any important information.

Name cleaning helper

The helper zavod.helpers.review_names makes it easy to

  1. prompt for proper name categorisation and cleaning
  2. get it reviewed

Once a dataset is fully reviewed, you can replace review_names() with zavod.helpers.apply_reviewed_names which will

  1. Call review_names() to do the cleaning and ensure a review exists
  2. apply each extracted name to the correct property of an entity if the review is accepted
  3. fall back to applying the original string cleaning wasn't deemed necessary, or human review is pending.

What's a clean name?

weakAlias

  • For Persons

    • single token e.g. Foopie or John but not John Smith.
    • Watch out for Chinese, Korean etc which don't have spaces - use an LLM or online translation to check namishness
    • Watch out Indonesian names can be single tokens. If in doubt, make it an alias

  • For organisations

    • acronyms of their name e.g. JSC SMZ for JOINT STOCK COMPANY SEROV MECHANICAL PLANT
    • really short short forms
    • names where a significant part is a really common term, e.g. TRO ITALIA or VA HOTLINE

previousName

  • Anything explicitly a previous name e.g.
    • formerly ...
    • f/k/a

alias

  • Anything explicitly an alias, e.g.

    • a.k.a
    • also ...
    • for Persons, when it's obviously a nickname, e.g. American Joe Miedusiewski
    • for Organisations, we might capture some really vague names as alias, especially when a more distinctive complete alternative name is known.

  • When variants are given, it's nice to expand the variants as aliases and keep the "primary" form, e.g. 

    - strings: ["Aleksandr(Oleksandr) KALYUSSKY(KALIUSKY)"]
  entity_schema: Person
  full_name: [Aleksandr KALYUSSKY]
  alias:
    - Oleksandr KALYUSSKY
    - Aleksandr KALIUSKY
    - Oleksandr KALIUSKY

name

  • Anything else that doesn't clearly need splitting or categorisation
  • If multiple names are given but not indicated to be aliases, treat them all as name.

Splitting

  • Transliterations given equal prominence can all be considered the same prop (if they're all in full) 
    - strings: ["Александар Добриндт / Aleksandar Dobrindt"]
  entity_schema: LegalEntity
  full_name:
    - Александар Добриндт
    - Aleksandar Dobrindt
  • Watch out for place names at the end - it might just denote a branch. e.g. these are different ways of saying Al-Qaida in Iraq so don't split the location at the end
    • The Organization Base of Jihad/Mesopotamia
    • The Organization Base of Jihad/Country of the Two Rivers
  • Especially in Person names, see if it's a cultural thing that's maybe one person's full official name
    • e.g. Amir S/O AHAMED means Amir son of Ahmed and appears to be one valid full name in Singapore

Prompt engineering

We use DSPy to write, optimise, and evaluate the prompt. The process is

  1. Ensure we have good example data in zavod/extract/names/dspy/single_entity_examples.yml
  2. Run zavod-tune optimise to find the ideal prompt for the data
  3. Run zavod-tune compare
    • This shows us how well the prompt works on the validation set
    • It also shows us how well it works directly, compared with via the DSPy client.

We use the prompt directly, rather than via DSPy, to avoid introducing DSPy as a production ETL dependency with significant additional dependencies. There is also a bug in leveldb which interacts with something in DSPy, which is a bit scary to have to dance around in production code.

Optimising the prompt

The GEPA optimiser in DSPy is used to develop an optimal prompt based on the example data and our feedback function zavod.extract.names.dspy.optimise.metric_with_feedback

Run it using

zavod-tune optimise

add --level light to use a subset of the data to experiment a bit more cheaply and quickly.

Be careful not to try to make the feedback function too fancy.

Big improvements have been made by just looking at the prompt, and identifying when the examples result in ambiguous or incorrect instructions in the prompt. First check that there aren't mistakes in the example data, and that there are enough examples of a scenario such that the randomly selected train, test, and validation sets have examples of a scenario to let GEPA develop instructions for all similar but different cases.

Examples take the form

- string: "Ch'oe Ch'o'l-min (a.k.a Choe Chol Min) (DPRK individual)"
  full_name: [Ch'oe Ch'o'l-min]
  alias: [Choe Chol Min]
- string: "Cho Yan Nathan; a.k.a Nathan Man Man"
  full_name: [Cho Yan Nathan]
  alias: [Nathan Man Man]

String represents the input string. The fields to extract are defined in zavod.extract.names.dspy.clean.CleanNamesSignature

The "optimised program" in DSPy speak is saved to zavod/extract/names/dspy/single_entity_program.json. This contains the prompt and some metadata.

Evaluate the prompt

Evaluate the optimised prompt by running

zavod-tune compare validation_results.json

Some progress information and overall statistics are printed, and details for each example are output to the provided JSON path.

...
DSPy score: 43.660000000000004 out of 47 (92.8936170212766%)
Direct GPT score: 39.284 out of 47 (83.58297872340425%)
Agreement: 35.0 out of 47 (74.46808510638297%)

We probably want to be careful not to let the Direct GPT score go below 80.

The scores aren't precisely a percentage, but 0 is given if none of the names are correct, 1 is given if all the names are correct, and partial correctness results in a score in between.

Agreement is when the same example results in precisely the same results via DSPy and directly.

Ideally add these lines to your commit message when you update the prompt.