Repairing Geometry During Python Imports

Problem Statement

A weekly shapefile arrives from a partner agency and a fraction of a percent of its polygons are invalid. Those few hundred features are the reason the validity audit keeps finding new work every Monday. This page moves the fix to where it belongs — the import boundary — and produces something the data owner can act on rather than a silent repair nobody sees.

Why the Naive Approach Fails

The usual import loop calls ST_GeomFromText or hands shapely geometry to GeoAlchemy2 and writes whatever arrives. Every invalid feature is written faithfully, and the pipeline reports success.

The second-most-common approach is barely better: wrapping every insert in ST_MakeValid. That silently rewrites the partner’s data on the way in, so the partner never learns their export is broken, and the repaired shapes drift a little further from the source every week.

Three import policies and what each one leaves behind Writing everything leaves invalid rows in the table and no signal. Repairing everything silently leaves a clean table but rewrites the source data with no record. Classifying leaves a clean table, a quarantine of genuinely broken features, and a report the upstream owner can act on. What the import leaves behind write everything table: 412 invalid rows signal: none the audit finds them next week, and the week after nothing improves repair everything table: clean signal: none source data silently rewritten every week the bug never gets fixed classify table: clean signal: a report quarantine holds the 11 features that are broken the source gets fixed The third column costs about forty lines of Python and is the only one where the failure rate goes down over time.

Production-Ready Implementation

Classification runs before any write and produces one of three outcomes per feature:

python
from dataclasses import dataclass
from enum import Enum

from shapely import wkb
from shapely.geometry.base import BaseGeometry
from shapely.validation import explain_validity, make_valid


class Outcome(str, Enum):
    ACCEPT = "accept"        # valid as supplied
    REPAIR = "repair"        # invalid but mechanically repairable
    REJECT = "reject"        # evidence of an upstream bug


@dataclass(frozen=True)
class Classified:
    outcome: Outcome
    geometry: BaseGeometry | None
    reason: str | None
    area_drift: float | None


# Reasons that mean "the exporter produced nonsense", not "the shape is messy".
FATAL = ("Hole lies outside shell", "Nested holes", "Too few points")

MAX_DRIFT = 0.01     # 1% of area


def classify(geom: BaseGeometry) -> Classified:
    if geom.is_valid:
        return Classified(Outcome.ACCEPT, geom, None, 0.0)

    reason = explain_validity(geom)
    if any(reason.startswith(f) for f in FATAL):
        return Classified(Outcome.REJECT, None, reason, None)

    fixed = make_valid(geom)
    if fixed.is_empty or fixed.geom_type not in ("Polygon", "MultiPolygon"):
        return Classified(Outcome.REJECT, None,
                          f"{reason} (repair produced {fixed.geom_type})", None)

    drift = (abs(fixed.area - geom.area) / geom.area) if geom.area else 0.0
    if drift > MAX_DRIFT:
        return Classified(Outcome.REJECT, None,
                          f"{reason} (area drift {drift:.2%})", drift)

    return Classified(Outcome.REPAIR, fixed, reason, drift)

The loader stages everything, then promotes in one transaction:

python
import psycopg
from psycopg import sql

def load(dsn: str, features, source_file: str) -> dict:
    counts = {o: 0 for o in Outcome}

    with psycopg.connect(dsn) as conn:
        with conn.cursor() as cur:
            cur.execute("CREATE TEMP TABLE staging (LIKE parcels) ON COMMIT DROP")

            with cur.copy(
                "COPY staging (source_id, name, geom) FROM STDIN (FORMAT BINARY)"
            ) as copy:
                for feature in features:
                    result = classify(feature.geometry)
                    counts[result.outcome] += 1

                    if result.outcome is Outcome.REJECT:
                        cur.execute(
                            """INSERT INTO import_quarantine
                                   (source_file, source_id, reason, raw_wkb)
                               VALUES (%s, %s, %s, %s)""",
                            (source_file, feature.id, result.reason,
                             wkb.dumps(feature.geometry)),
                        )
                        continue

                    copy.write_row(
                        (feature.id, feature.name, wkb.dumps(result.geometry))
                    )

            # promote everything staged, in one atomic step
            cur.execute("""
                INSERT INTO parcels (source_id, name, geom)
                SELECT source_id, name, geom FROM staging
                ON CONFLICT (source_id) DO UPDATE
                    SET name = EXCLUDED.name, geom = EXCLUDED.geom
            """)
        conn.commit()

    return {o.value: counts[o] for o in Outcome}

Because the staging table is temporary and the promotion is a single statement, a failure anywhere leaves the live table exactly as it was — while the quarantine rows, written on their own connection-level transaction boundary, survive to explain why.

Where each class of feature ends up Twelve thousand incoming features split three ways. Eleven thousand five hundred and eighty are accepted unchanged, four hundred and nine are repaired, and eleven are quarantined. Accepted and repaired features go to the staging table and are promoted atomically; quarantined ones go to a review table with the reason and the original bytes. One file, three destinations source file 12,000 features classify() no writes yet accepted — 11,580 unchanged repaired — 409 drift under 1% quarantined — 11 with reason and raw WKB parcels one atomic promote The eleven quarantined features are the report. Send it upstream weekly and the number goes down.

Reprojection belongs in the same pass

Imports rarely arrive in the SRID the table uses, and reprojection interacts with validity in a way that is easy to get backwards. Transform after repair, not before: a self-intersecting polygon reprojected through PROJ stays self-intersecting, but its coordinates have moved, so the repair now operates on numbers that no longer match the source file and the drift comparison loses its meaning.

python
from pyproj import Transformer
from shapely.ops import transform as shapely_transform

_to_4326 = Transformer.from_crs("EPSG:25833", "EPSG:4326", always_xy=True).transform

def prepare(geom):
    """Repair in the source CRS, then reproject — in that order."""
    result = classify(geom)
    if result.outcome is Outcome.REJECT:
        return result
    return Classified(result.outcome,
                      shapely_transform(_to_4326, result.geometry),
                      result.reason, result.area_drift)

Creating the Transformer once at module scope rather than per feature matters more than it looks: constructing one builds a PROJ pipeline and costs milliseconds, so a per-row construction turns a two-minute import into an hour. The same applies to ST_Transform in SQL, where PostGIS caches the pipeline per session — another reason to reproject in one set-based statement rather than row by row.

Configuration and Tuning Knobs

The FATAL reason list is the policy. Start with the three listed above — a hole outside its shell, nested holes and too-few-points are all coordinate bugs rather than messy geometry — and add to it whenever a repaired feature turns out to have been wrong. Every addition should be traceable to an incident.

MAX_DRIFT decides how much reinterpretation counts as repair rather than replacement. One percent works for parcels. Building footprints deserve a tighter bound; a coarse habitat polygon can tolerate more.

The staging approach costs one extra copy of the incoming batch. For a twelve-thousand-feature weekly file that is nothing. For a hundred-million-row initial load, stage in chunks of a few hundred thousand and promote each chunk, so a failure late in the load does not discard everything.

The number that should go down Quarantined features per weekly import across twelve weeks. The rate sits near forty per week until the report reaches the data owner in week five, then falls steadily to near zero by week ten as the upstream export is fixed. Quarantined features per weekly import report sent upstream 40 0 wk 1 wk 12 A repair pipeline whose quarantine count is flat over months is repairing the same bug every week. The falling line is the whole justification for reporting rather than silently fixing.

Verification Steps

sql
-- nothing invalid entered the live table from this run
SELECT count(*) FROM parcels
WHERE NOT ST_IsValid(geom)
  AND updated_at > now() - interval '1 hour';

-- the quarantine, grouped for the report to the data owner
SELECT reason, count(*) AS features, min(source_id) AS example
FROM import_quarantine
WHERE source_file = '2026-W32-parcels.shp'
GROUP BY reason
ORDER BY features DESC;

The second query is the deliverable. It names the failure, counts it and gives an example id the partner can look up in their own system — which is what turns a recurring cleanup into a fixed bug.

Gotchas Checklist

  • shapely.make_valid and ST_MakeValid are the same GEOS call but different defaults. Shapely’s make_valid uses the linework method unless you pass method="structure" on shapely 2.1 or newer; PostGIS defaults the same way. If your SQL repairs and Python repairs disagree, this is why.
  • explain_validity returns a human string, not a stable code. Match with startswith on the leading phrase as shown, never with equality — the coordinate suffix in the message differs per geometry.
  • Quarantine the original bytes, not the repaired shape. The point of the table is to reproduce the problem; a repaired copy cannot do that.
  • ON COMMIT DROP temp tables are per-session. If your loader uses a connection pool and hands the connection back between the copy and the promote, the staging table vanishes. Keep both on the same explicit connection, as above.
  • Do not let the quarantine grow unbounded. Add a retention policy — ninety days is generous — or it becomes a second copy of every bad file you have ever received.