I've seen the committers thread on trying to improve dataclasses' start time performance but it seemed focused on the execution time, while that is an area to look at I think it may be worth looking at improving the import time of dataclasses itself.
I've been working on my own implementation of the same idea and had
found that importing some stdlib modules had a significant impact on the overall time it took to run. On looking at
dataclasses I noticed that it also imported some of the same modules.
Dataclasses imports a few fairly slow stdlib modules to perform single tasks in areas which may never be needed by some users.
inspect is imported to populate a docstring for a class if one doesn't exist1.
Removing this import saves nearly half the time it takes dataclasses to import.
I think this docstring could probably be written using the information in fields or just delayed unless needed but for the sake of
this quick test I've just commented it out.
re is imported in order to handle string annotations in the case that they are one of the 'special' types
(ClassVar, InitVar, KW_ONLY). This might never be needed so the import can be delayed until it is.
functools is imported solely to wrap an inner function for _recursive_repr.
_recursive_repr is defined in the dataclasses module to avoid a dependency on reprlib.
However, functools depends on reprlib (in fact, it imports recursive_repr itself).
Importing recursive_repr from reprlib is faster than importing functools.
copy is imported to be used in the asdict and astuple methods. Delay these imports until the methods
are used.
Note that inspect depends on re and re depends on functools so there's no benefit to removing
functools without removing inspect and re and no benefit to removing re without removing inspect.
copy is independent.
All tests are done with python 3.11.1 on a Dell XPS13 laptop running ubuntu 20.04.
These are the results of timing basic import of dataclasses with imports gradually removed.
dataclassesis the original, unaltered module as of 3.11.1dataclasses_no_inspecthas theinspectimport removed and the use commented outdataclasses_no_rehas thereimport deferred until the compiled regex is needed. It also hasinspectremoved.dataclasses_no_functoolshas thefunctoolsimport replaced with thereprlibimport, also noinspectorre.dataclasses_no_copyhas all of the previous imports removed, along withcopy
python -X importtime -c "import dataclasses"
import time: self [us] | cumulative | imported package
import time: 899 | 899 | _io
import time: 153 | 153 | marshal
import time: 1738 | 1738 | posix
import time: 1811 | 4600 | _frozen_importlib_external
import time: 554 | 554 | time
import time: 639 | 1193 | zipimport
import time: 203 | 203 | _codecs
import time: 1263 | 1466 | codecs
import time: 1144 | 1144 | encodings.aliases
import time: 2906 | 5514 | encodings
import time: 703 | 703 | encodings.utf_8
import time: 323 | 323 | _signal
import time: 148 | 148 | _abc
import time: 456 | 604 | abc
import time: 699 | 1303 | io
import time: 127 | 127 | _stat
import time: 200 | 327 | stat
import time: 2057 | 2057 | _collections_abc
import time: 76 | 76 | genericpath
import time: 169 | 244 | posixpath
import time: 887 | 3514 | os
import time: 173 | 173 | _sitebuiltins
import time: 1320 | 1320 | _distutils_hack
import time: 232 | 232 | sitecustomize
import time: 1725 | 6962 | site
import time: 1015 | 1015 | types
import time: 206 | 206 | _operator
import time: 833 | 1038 | operator
import time: 192 | 192 | itertools
import time: 327 | 327 | keyword
import time: 344 | 344 | reprlib
import time: 122 | 122 | _collections
import time: 1623 | 2607 | collections
import time: 128 | 128 | _functools
import time: 1078 | 3811 | functools
import time: 2354 | 8217 | enum
import time: 85 | 85 | _sre
import time: 375 | 375 | re._constants
import time: 1321 | 1695 | re._parser
import time: 145 | 145 | re._casefix
import time: 458 | 2382 | re._compiler
import time: 205 | 205 | copyreg
import time: 726 | 11528 | re
import time: 343 | 343 | _weakrefset
import time: 497 | 839 | weakref
import time: 99 | 99 | org
import time: 20 | 119 | org.python
import time: 23 | 142 | org.python.core
import time: 262 | 1242 | copy
import time: 1372 | 1372 | _ast
import time: 672 | 672 | contextlib
import time: 1166 | 3209 | ast
import time: 163 | 163 | _opcode
import time: 517 | 680 | opcode
import time: 931 | 1610 | dis
import time: 194 | 194 | collections.abc
import time: 321 | 321 | warnings
import time: 181 | 501 | importlib
import time: 74 | 575 | importlib.machinery
import time: 225 | 225 | token
import time: 1098 | 1323 | tokenize
import time: 197 | 1519 | linecache
import time: 1964 | 9068 | inspect
import time: 1070 | 22907 | dataclasses
python -X importtime -c "import dataclasses_no_copy"
import time: self [us] | cumulative | imported package
import time: 613 | 613 | _io
import time: 147 | 147 | marshal
import time: 1416 | 1416 | posix
import time: 1334 | 3508 | _frozen_importlib_external
import time: 409 | 409 | time
import time: 452 | 860 | zipimport
import time: 318 | 318 | _codecs
import time: 1686 | 2004 | codecs
import time: 2241 | 2241 | encodings.aliases
import time: 3293 | 7537 | encodings
import time: 882 | 882 | encodings.utf_8
import time: 428 | 428 | _signal
import time: 104 | 104 | _abc
import time: 490 | 594 | abc
import time: 621 | 1215 | io
import time: 162 | 162 | _stat
import time: 244 | 406 | stat
import time: 1925 | 1925 | _collections_abc
import time: 79 | 79 | genericpath
import time: 162 | 240 | posixpath
import time: 895 | 3466 | os
import time: 204 | 204 | _sitebuiltins
import time: 1337 | 1337 | _distutils_hack
import time: 252 | 252 | sitecustomize
import time: 1731 | 6987 | site
import time: 541 | 541 | types
import time: 272 | 272 | keyword
import time: 176 | 176 | itertools
import time: 558 | 558 | reprlib
import time: 1296 | 2840 | dataclasses_no_copy
Test command ('pass' is included to show base python start time).
hyperfine --warmup 10 --runs 100 -N --export-markdown=dc_import_tests.md \
'python -c "pass"' \
'python -c "import dataclasses"' \
'python -c "import dataclasses_no_inspect"' \
'python -c "import dataclasses_no_re"' \
'python -c "import dataclasses_no_functools"' \
'python -c "import dataclasses_no_copy"'
Result:
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
python -c "pass" |
11.3 ± 0.3 | 10.9 | 12.2 | 1.00 |
python -c "import dataclasses" |
30.2 ± 0.6 | 29.2 | 31.5 | 2.68 ± 0.09 |
python -c "import dataclasses_no_inspect" |
20.3 ± 0.5 | 19.7 | 22.8 | 1.80 ± 0.06 |
python -c "import dataclasses_no_re" |
16.1 ± 0.3 | 15.6 | 16.9 | 1.42 ± 0.04 |
python -c "import dataclasses_no_functools" |
14.0 ± 0.3 | 13.6 | 14.9 | 1.24 ± 0.04 |
python -c "import dataclasses_no_copy" |
12.8 ± 0.5 | 12.4 | 15.1 | 1.13 ± 0.05 |
Footnotes
-
This was true in Python 3.11 but in Python 3.12 it is also used to get the annotations. ↩