Design Decisions
Concrete Dependencies and Updating
This package, like many others, has dependencies that need to be fulfilled for
proper functionality to be satisfied.
While these general requirements, found in the pyproject.toml file, are
known as “abstract” requirements, we also choose to codify “concrete”
requirements here via the poetry.lock file.
This distinction is described well by [Stufft], using the setuptools-based
resources which are parallel in use to the above respectively described files.
While xaitk-saliency is a library, we choose to retain concrete
dependencies via the poetry.lock file to maintain consistency
of environment across developers as well as CI processes.
This falls in conceptual line with the “Developing Reusable Things or How Not
to Repeat Yourself” section from [Stufft].
However, we are still a “library” and desire to make sure that we work with the
“latest” versions of our listed abstract dependencies (within some reasonable
time window).
Currently, such concrete “version bumps” happen in the form of periodic update
branches that update the poetry.lock file via a poetry update call.
These updates are submitted as PRs to the upstream repository and allow the
standard suite of CI checks to be performed to make sure the updated versions
do not break anything.
The timing of such updates are currently not concretely scheduled, nor are they
specifically tied to events, but more on an “every so often” cadence that is
relatively more frequent than versioned releases.
Image Format
We choose to use the numpy.ndarray data structure for our image
representation in this toolkit.
Earlier, we utilized the Pillow package’s PIL.Image.Image data structure
but encountered issues in certain use cases regarding large images, images with
non-standard quantities of channels (e.g. > 3) or with imagery consisting of
12 or 16-bit valuation.
Additionally, other popular and highly utilized packages in the Python
community, like OpenCV, Scikit-Image, and PyTorch to name a few, utilize raw
numpy.ndarray matrices as the container for image data.
Import Guards
Some xaitk-saliency implementations are conditional and need an optional dependency (an
“extra”). The class is published from a public module and reported under that path, never
under the private module that defines it. A generated default config (make_default_config) records
xaitk_saliency.interop.maite.object_detection.MAITEDetector
rather than
xaitk_saliency.interop.maite.object_detection._model.MAITEDetector
xaitk_saliency._guard.guard() provides all of this functionality. Every module that exposes an
implementation (needing an extra) calls it exactly once; apart from the if TYPE_CHECKING: block and
the imports the call itself needs, the module does nothing else at import time.
from collections.abc import Callable
from typing import TYPE_CHECKING, Any
from xaitk_saliency._guard import Group, guard
if TYPE_CHECKING:
from xaitk_saliency.interop.maite.object_detection._model import MAITEDetector as MAITEDetector
__getattr__: Callable[[str], Any]
__dir__: Callable[[], list[str]]
__all__: list[str]
__getattr__, __dir__, __all__ = guard(
namespace=globals(),
groups=[
Group(
symbols={"MAITEDetector": "xaitk_saliency.interop.maite.object_detection._model"},
extras=["maite"],
),
],
)
Terminology
- package
A directory with an
__init__.py. A public package is one users import from; a private one starts with an underscore and exists only to hold leaf modules.- leaf module
The
.pyfile that actually defines a class, and whatsymbolspoints at. Never a package. xaitk-saliency’s leaves are private modules inside a public package. Usually that package’s__init__.pycallsguard()(xaitk_saliency.interop.maite.image_classificationfronts its_model); the other layout is a public module fronting a sibling leaf, the wayxaitk_saliency.utils.cocofrontsxaitk_saliency.utils._coco.- submodule
A module named relative to the package containing it, which is why
guard()takessubmodules=. Use it only when the parent is the point; otherwise say module.- group
One
Group: the symbols that share the same set of requirements. Addextras=as the requirements need. Multiple extras is still one group —extras=["maite", "tools"]means both are required, not that two groups are needed. A module declares as many groups as it has distinct sets of requirements.
Adding an implementation
Add the class to the symbols of the group with matching requirements or create
a new group, if existing ones do not satisfy requirements, and add it to the if TYPE_CHECKING: block.
symbols maps the public name to the leaf module. Write that path out in full;
hoist a shared prefix into a _CONSTANT only when the line would otherwise run
past the 120-character limit.
The if TYPE_CHECKING: block is not decoration. __getattr__: Callable[[str], Any] makes pyright
treat every attribute of a guarded module as Any, so a symbol missing from the
block still imports, still type-checks, and still scores 100% on
--verifytypes — it just silently loses its signature, and with it: argument
checking at every call site, @override resolution, and editor hover.
tests/test_guard_declarations.py::test_type_checking_block_matches_the_declaration
is the only thing that catches it, in either direction: a declared symbol missing
from the block, or a leftover in the block that is no longer declared. The bare
annotations above the guard() call matter too:
__getattr__: Callable[[str], Any]
__dir__: Callable[[], list[str]]
__all__: list[str]
--verifytypes requires a declared type for every exported module-level symbol.
Without these it reports __getattr__ and __dir__ as ambiguous rather than
known, which drops the score below 100% and fails. __all__ is not counted
either way; it is declared alongside them so the three read as one unit.
Describing extras
extras is a list of all dependencies required for a Group; a nested tuple indicates a
choice between two or more alternatives, i.e., one of the tuple alternatives must be available at runtime.
extras=["maite"] # maite
extras=["maite", "tools"] # maite and tools
extras=[("a", "b")] # a or b
extras=["c", ("a", "b")] # c, plus either a or b
The guard renders these requirements into the error that the user sees, with one pip install per
possibility of meeting the requirements. Requirements are specified as a list rather than a string because
there is no pip syntax for the “and either of these” situation that nested tuples represent.
Registering modules for discovery
Being importable is not being discoverable. Register the module that calls
guard() and advertises the class — the same public path the class is
published under — never the leaf module and never a parent that only
re-exports.
xaitk-saliency has no example of this rule yet.
Rules
Two things the guard cannot check for you, because both are decisions you make when you declare a group.
Point
symbolsat a leaf module, never at anything that runs its own guard.# Right: the leaf module that defines the class. "MAITEDetector": "xaitk_saliency.interop.maite.object_detection._model" # Wrong: a public package, which runs its own guard() call. "MAITEDetector": "xaitk_saliency.interop.maite.object_detection"
Pointing at a public package would require that package’s own
guard()call to resolve first, reintroducing the import-order dependence the guard exists to remove.A private package’s
__init__.pyholds nothing but a docstring. Re-exporting an extras-dependent leaf creates a second, unguarded path to the class and breaks the compliance walk (tests/scripts/guard/packages_import_without_extras.py, run bytests/test_import_guards_e2e.py) that imports every package with no extras installed.
What the guard guarantees
There is nothing to do about the following — it is what guard() is for, and no
declaration you can write will break it. It is recorded here because it explains
why the guard behaves the way it does.
__dir__ never advertises a name that __getattr__ would raise on. SMQTK’s
plugin discovery calls getattr on every name dir() returns and catches
nothing, so a single raising name would abort the entire discovery pass — every
implementation after it going undiscovered, not just the one that raised.
_Guard.dir returns __all__, and __all__ only ever gains symbols that
actually resolved, so that state cannot arise.
Holding that guarantee is why symbols resolve eagerly, at guard() time: whether
an extra is installed cannot change mid-process, so a group whose extra is missing
is never advertised in the first place. Only declared submodules are imported
lazily, on first access.