Continuation of discussion from https://github.com/eslint/eslint/issues/15453

Discussion on ESM-only projects.md

I understand your argument about force, and although I very much sympathize with the general tendency to seek to do things without what one could in a sense call "force", I think there are a few factors which are obscuring matters here.

For one, I think we have to be careful that we are not engaging in a fallacy of equivocation. While it is especially disconcerting to see "force" being used ot justify even well-intended behaviors, when understood as compulsion under threat of violence or imprisonment by the state, obviously we are not talking about anything remotely similar in degree here (assuming all would even agree that this is indeed a form of compulsion at all).

So saying "it's always wrong to force things on people" might be seen as gaining credence by attacking the strawman of a supposed claim justifying physical compulsion. This brings me to my second point which is about the polemical-sounding nature of engagement with such a fallacy.

Polemicists use ambiguous language such as this to obscure matters in the course of seeking to win temporary support for their arguments, but their doing so also obscures a detached search for truth to really look dispassionately together at what is at hand without complicating things with language that may be interpreted in a way that evokes more moral outrage than is relevant.

In this vein, to highlight the polemical impression such an argument gives from the other side, suppose I were to have used similarly equivocal language to suggest that you were in fact "forcing" others. I am not supporting this kind of polemical language, because I don't find such strong terms helpful, but it may incidentally also highlight the point about polemicism as well as the logical points about ESM breakage that I wish to make.

A polemicist might claim that you were actually the one being forceful on insisting on your viewpoint or that you were seeking to "force" the project to maintain backward compatibility. Again, this would be unfair, given the strong connotations of physical pressure of "force", but I expect you can admit, you are indeed seeking to exert some "force" in a weaker sense. True, you are not holding a gun to our head, but neither are developers of ESM-only projects. ESM-only projects are simply not doing work that you want them to continue doing.

While you might respond that there is some kind of social contract calling on us to help others, so this "burden" on us was merited, even setting aside your categorical rejection of "force" in pressuring us, still, why are we not part of the social contract too? Why are the needs of the many of us who don't want to be stuck with extra procedures not relevant? Is our free labor here a kind of slavery (yes, more potentially polemical language) whereby we are compelled to do your bidding rather than our own? This is not to mention some of us not wanting to prolong the community having no incentive to get around to supporting ESM--what about that social contract (to "think of the children" if I may further the polemicism)? If there were a technology that could speed up the Internet by 100 times, and yet it required 1 hour of pre-announced internet down-time, "forcing" users of the internet to suffer for the benefits of a new innovation, would you not be at least open to considering the idea? What if it were 1000 times the speed and only five minutes of down-time. Surely, you can see there is a spectrum here.

But my main goal is to set the polemical discussions of "force" aside, and instead get back to suggest in non-charged language that one might argue that you are putting the burden you wish to seek to avoid instead on ESM-only-favoring developers and future generations due to some absolute, unyielding value being assigned to backward compatibility.

Now backward compatibility is an understandably admirable standard to strive for, especially in fields like the web where there is such a vast prior output (and by many not so technical users) that one would not wish to lose access to it).

But another factor I would like to argue, unrelated to polemics, is that this form of "backward compatibility" is akin to the type of breakage we see as in "breaking the web". This can hardly be compared in degree to purposely creating a browser which no longer is capable of displaying old websites. We're talking about competent developers having the inconvenience of needing to tweak their code (and who regularly need to make tweaks anyways, e.g., to avoid vulnerabilities or support their userbase), and yes, perhaps face some suffering as the dependency chain is fixed or alternatives are found. But the changes required are not extreme in degree, even if it is admittedly of no small consequence either. The impossibility of contacting all website owners to "fix" their code is not the same as contacting all non-ESM projects to retool their support for ESM. In addition, most ESM projects today can continue using non-ESM code, so the burden is really only on non-ESM or polyglot dependees rather than dependents. As developers start getting used to creating ESM code, they will make ESM-friendly or polyglot versions--because they wish to support the community, and the problem can be assisted from the ground up.

I mean, obviously this is open source - everything's "as is", everyone can do whatever they want. That doesn't mean they're free from the consequences of their actions, including how others choose to label them.

I think that it's important to consider harm reduction as the primary factor here. The harm caused by dropping backwards compatibility is objectively larger - in that it makes usage impossible for nonzero human beings (it is simply not possible to safely transform code one did not author; the JS language doesn't permit it) - than the harm caused by those who wish to embrace the bleeding edge being "required" to employ trivial build tools to do so. In other words, a package going ESM-only makes its use impossible for some (probably most) — but a package staying CJS-only, or going dual mode (which rarely offers any value over staying CJS-only) causes: 1) the maintainer has to avoid shiny new syntax, or, has to use a transpiler; 2) non-node ESM consumers (since native node ESM can import CJS perfectly fine) will have to use a build tool (like they're doing already; minifiers count) to transform the code before using it on the non-node target. I can not comprehend an argument that says that "impossible" is less damaging than "inconvenient".

Stepping away from the "force"/polemical/ethical discussion:

---

There's not actually any benefit to choosing native ESM over CJS.

Some might offer "static analyzeability" - but both CJS and ESM can be used statically and dynamically, and both can be trivially constrained to static forms with a linter, as the most popular styleguides already do.

Some might offer "tree-shaking" - but CJS is precisely as tree-shakeable as ESM. It is true that most bundlers have imo shirked their duty and only made it easy to do in ESM, but there's no technical reason it's not doable - this means that effectively, only ESM is tree-shakeable, fair enough. However, tree-shaking is only a half measure to reclaim resources that were sloppily overconsumed - it can never do as good a job as "only import/require what you need in the first place". If you do that (ie, deep imports, over manifest/barrel exports), then tree-shaking offers no benefit anyways, and that argument disappears.

Some might offer "but progress!", but just because new syntax arrives does not mean using it is automatically progress. In the browser, native ESM is slow by design (due to waterfall loading of dependencies), and I'm not yet aware of a standard browser way to address that (but it'd require a build process, of course). In node, native ESM lacks many fundamental tools, many - but not all - of which loaders will supply, whenever they're ready (which they aren't yet). It also lacks lots of ecosystem support, altho that will of course catch up (my feelings on ESM don't change that tools still need to support it, including my own packages).

In other words, if you can show me a package that has gone ESM-only, and a justification of why that was a better choice than authoring in ESM and adding a build process (and "but i wanna" or "progress!" isn't sufficient), I'd be very fascinated to learn about it.

I mean, obviously this is open source - everything's "as is", everyone can do whatever they want. That doesn't mean they're free from the consequences of their actions, including how others choose to label them.

Sure, though I personally am not a fan of labeling people, except perhaps for the persistently and deliberately malicious or deceptive, and even then, solely for protection of others. I think people are more willing to contemplate issues without labels hanging around them in a negative context. I mean it can sometimes be helpful to refer to tendencies as "liberal" or "conservative", for example, but the terms being used in a hostile context do not conduce to people of good will objectively examining a question with justice. We certainly don't need such firmly affixed or harsh labels of people in the open source world either (though labeling behaviors is a different matter).

I think that it's important to consider harm reduction as the primary factor here. The harm caused by dropping backwards compatibility is objectively larger - in that it makes usage impossible for nonzero human beings (it is simply not possible to safely transform code one did not author; the JS language doesn't permit it) - than the harm caused by those who wish to embrace the bleeding edge being "required" to employ trivial build tools to do so.

I think that is a very fair question and generally objective representation, though instead of "harm reduction", I think it would be more objective to a fuller discussion to characterize it as a question of maximizing the benefit/cost ratio since one ought to take into account benefits as well (even if not treating all costs laid on users as equally offset by some marginal benefits). When allowing for raised speed limits, for example, it is simply the case to be likely that fatalities will increase, yet societies support the (non-individually-targeted) distribution of the high cost in favor of the benefits. Society cannot only function based on harm avoidance. Even if one thinks of it in terms of the "harm" of not being able to get to work as expeditiously, it is kind of a negative focus, so I think more illuminating to simply speak to it as being about a balance between both.

There is one other slight benefit to mention, even though largely just another side of the same coin. In all likelihood, ESM-only distributions would lead more quickly to the source code also being moved away from CJS, thus allowing for a uniform standard across environments in source. Beginners have just one syntax to learn, tooling related to querying AST is simplified, etc. I'll admit this is probably even less consequential than reducing build steps per se (since build steps are not quite marginal, as the freedom to quickly experiment and deploy can lead to more projects or proofs-of-concept getting off the ground). But not entirely negligible. (See the next block on minimization.)

In other words, a package going ESM-only makes its use impossible for some (probably most) — but a package staying CJS-only, or going dual mode (which rarely offers any value over staying CJS-only) causes: 1) the maintainer has to avoid shiny new syntax, or, has to use a transpiler; 2) non-node ESM consumers (since native node ESM can import CJS perfectly fine) will have to use a build tool (like they're doing already; minifiers count) to transform the code before using it on the non-node target.

The rest is fair, I think, but one doesn't need minifiers for quick demos of and experiments with smaller-size libraries (something I find myself working on and with fairly often).

There's not actually any benefit to choosing native ESM over CJS...

I think you mean technical benefits given the above-mentioned. But sure, to the rest of your comments, your arguments appear reasonable to my however limited ears.

I can not comprehend an argument that says that "impossible" is less damaging than "inconvenient".

Though I think "impossible" is a bit too strong as the developer community does tend to submit PRs, make forks, or as needed start from scratch, though I'll concede it is a rather high cost, especially for a pretty negligible benefit. So I think this all gets us much closer to the real crux of the issue.

If a society costs engages in changes which are acute for some individuals and industries and the benefits are thin and widely distributed, such as with free trade, this will be hotly contested as it is.

And here, we are even admittedly talking about widely distributed costs, making for a change that is indeed less likely to be warmly received, especially in the short term.

So indeed it is possible that like VHS and Beta-max, the community will simply reject the drive to the superior outcome due to the cost and things will carry on as usual.

However, developers tend, I think, to be more conscientious, seeking to stay ahead of the curve, preferring the "ideal" solution even if at some cost to themselves. I'll concede that ESM-only is no super strong selling point, but developers also do like a sense of cleanness as well as practicality.

In other words, if you can show me a package that has gone ESM-only, and a justification of why that was a better choice than authoring in ESM and adding a build process (and "but i wanna" or "progress!" isn't sufficient), I'd be very fascinated to learn about it.

I don't think this is something to be highlighted on a case by case basis, especially if looking merely at the present time. I mean the marginal benefits mentioned do exist in some sense now, but it is likely to suffer from an adoption cost as well, so practically its evident net benefits aren't anything to speak of now.

I think the implementers now are more acting from idealism in wanting to see adoption--much less dramatically so, but not too unlike the developers in the dark ages of IE's stagnation over the open web who resisted its lack of progress by making apps which left out IE support, not so much out of spite perhaps as for convenience in being able to build quick demos, or non-commercial sites that way (even if coming at a cost of reducing uptake) and at times a touch of idealism. But this idealism surely had its impact in leading us to drop off the cruft and lack of standard behavior in needing to target both IE and the rest, toward just being able to target one standard, except when it comes to cutting edge features still under development.

So the benefits I think in this case are simply that such actions can prod changes for the sake of tomorrow, albeit admittedly with pain in the here and now, but which can lead ultimately to however small perpetual ongoing benefits into the future, leading to a single format--a format which is inarguably better in being standard, uniform across environment, and ultimately fostering a minimization in build steps.

I concede that the cost is quite high relative to the benefits, however, and it will probably be resisted by most until the costs of not refactoring become too high and until critical mass makes it easier to use ESM exclusively.

While one could well argue that encouraging dual package support would lessen the pain of transitioning, I think this will be resisted because:

1. Existing CJS projects often do not like to add to their build complexity or distribution size, especially at a time when their users are not clamoring en masse for ESM distributions. Idealists might change, and the agreeable, but that's all.

2. Many remain unaware of the benefit I spoke of earlier that one can still use ESM as is in browser demos, or certain other non-Node environments, so even those which use ESM in source may be disinclined to provide an ESM distribution, causing a hassle for ESM users akin to a popular project not providing binaries--you have to figure out how to roll it up/pack it yourself.

3. As you say, even ESM-only idealists might use CJS dependencies. This is an advantage for easing the pain of transitioning, but that assumes net efforts are in fact being made to transition.

Projects with mixed dual ESM/CJS-supporting dependencies are likely to do little to anything to conduct the prodding necessary to see a transition take place for projects away from CJS in source and/or toward ESM-only distributions to get to the benefits desired, especially if even new projects feel it is simpler to just use CJS to do their own avoidance of a build step if it comes at no cost. As with moral hazard, there is little incentive to change for most. Dual support projects with ESM in source may at least encourage familiarity with ESM, but they don't move the needle away much for projects continuing to use CJS only (and they don't give the benefits of avoiding a build step unless the source is a self-sufficient distribution file itself).

Albeit yes, with exaggeration of the benefits at the outcome, I think ESM-only projects are a bit like no-smoking restaurants at the beginning of such movements. They will annoy customers even at some cost to themselves, but they cause notice of the issue and increase the likelihood of change. Unlike restaurants with no-smoking sections, they provide an entirely smoke-free environment even if non-welcoming to smokers. Yes, smoke is more clearly a harm, but one can plausibly express the lack of a cross-environment standard as a "harm" too, as it is a cost relative to the ideal.

In short, I don't deny that the benefit-cost ratio is very low (unless taking into account the benefits accruing indefinitely after ESM-only becomes ubiquitous), but I think there is room for the idealists and the more neutralish dual-packages to collectively create a positive result in the end, especially if another kind of idealist is willing to sacrifice in making PRs for at least switch to ESM in source and distribution and others are open to the changes, but it will really be the ESM-only packages that prod change, I think.

But just to be clear; my point is not to speak to arguments of ESM being an inherently superior format (e.g., relative to static analysis). My points would be the same if the standard adopted for the browser and ES had instead been CJS. There would be just one standard.

Websites dropping support for IE < 9 didn't kill those browsers - the community migrating did. Similarly, individual package authors will either be premature, and have no positive impact - or the ecosystem will bring them along. In other words, your smoking section analogy applies: smoking in restaurants didn't die because restaurants started removing smoking sections, it died because nobody wanted to taste cigarette smoke in their food. Restaurants didn't start the trend - they waited until it was a good business decision to omit smoking sections, and followed it.

What I'm hearing is that you agree with me that ESM-only package authors have no technical argument for using it, only that they're trying to force the ecosystem along a path that they think is superior? (we don't have to agree on which outcome, or path, is actually superior)

Yes, re: non-smoking, you are right, surely that is the general underlying reason for success either way, and I shouldn't have oversimplified it like that. But even individual business owners are not like an aggregate of amoral shareholders holding no emotions either way. They might tend to act rationally, but there are a good number of businesses, especially perhaps among the first adopters (even if not all first adopters act for such reasons) who say, "Yeah, this is something we need to do something about; I lost my mom to cancer, etc., and maybe there are people who want to avoid the smoke too", etc. (and some would have resisted the commercial benefits because they liked smoking or being open to smokers.) But still, there is a mechanism by which "force" leads to a positive outcome.

But to the main point, I'm not as qualified to speak to all the technical merits, and your arguments sound convincing, but yes, my own rationale hasn't been so much in relation to the technical merits anyways. The "superiority" solely relies in one being a single standard which has the standards-based momentum behind it to become the one way.

(The static analyzability argument in your scenario which can only be circumvented with linting does require some tooling to just work and although I personally can't bear a project without linting, I do think JavaScript should have a syntax that just works without such steps being required, but to my understanding, if CJS were a candidate for browsers, static analyzability could, I imagine, have been imposed on a subset of CJS.)

If there were a possibility on the table for browsers to support CJS and convince everyone to support that, and it could be done without the chaos it would presumably cause in confusability with bundles and such, I would have personally been fine with CJS becoming the standard instead. But unless you are trying to argue for CJS being used natively in browsers (so that there can be a different but also single standard around which the community could eventually coalesce without a build step), being told to just use browserify or webpack or whatever doesn't satisfy me for my rationales since the goal is to be both buildless as well as use a single syntax cross-environment.

So, the idea from my perspective is to reach to the benefits of being able to use just one standard. Copy paste any good, non-environment-specific JavaScript code of interest off the web, and it will just work. Import from npm and it will (eventually) just work. Simplify linting routines by not having to configure the environment for simple, poyglot scripts. That's the future I'd like to reach for such a fundamental and all-too-long delayed building block of the great, and potentially greater, JavaScript ecosystem. I fully admit it is not the most urgent need out there, but I would like us to get there, and think we can with a mix of some prodding and some gradualism. Distractions related to build steps adds a stumbling block toward new code being quickly experimented with. When I'm trying something out, in the browser or Node, at the early stages, I just want to be able to add an import statement, and write code modularly without worrying about build steps until the performance trade-off becomes noticeable. I also don't want to have to write README's with separate subsections on import methods per environment, build details, etc., so as to be helpful for newcomers. I just want to put in the effort so it can become more simple.

Because dynamic import exists, ESM also requires linting to preserve static analysis.

Gotcha.

The utopia you want simply can’t ever happen. Browsers have a DOM, node/deno can make network requests and talk to a filesystem and have env vars etc. environments are different, and without a build process, there’s a huge amount of code that simply isn’t universal.

You always have to know what environment(s) code is written for, and you only sometimes can transform it to work in other ones.

Node has JSDom, browsers can be empowered through the File Access API to talk to the file system, use node-fetch or file-fetch, reuse URLSearchParams, etc. I live almost every day in other regards in such a very comfortable utopia with various projects that are environment-independent and can leverage libraries that work in different environments. That's one of the biggest draws of JavaScript to my taste and to others I know as well. With linting especially--and I wasn't speaking against linting earlier, merely that it is ideal not to have to depend on it--and for ESM or CJS one doesn't need to depend on linting just to use it even if certain errors may get through without accurate use for a given purpose as is expected--one can prevent use of globals to ensure that any environment-specific code goes into its own entry file.

And based on what I've encountered, many projects are open to becoming out-of-the-box environment-independent, but their author just happened to write it in CJS...

CJS that happens not to use environment-specific features is precisely as environment-independent as ESM that happens not to, since both are well-understood input formats.

Since import maps aren't a standard, there remains no way to have node code with nonzero dependencies work in a browser without a build tool. If a build tool is required, ESM and CJS can both be identically consumed.

In other words, the only time I think you have a reasonable argument for environment-independent usage is when you have code that meets all these requirements:

1. exists in a single file
2. does not import or require anything
3. does not use any environment-specific features (filesystem, network, user input, DOM)
4. does not use any APIs outside of the main language specification, which excludes Intl - i can make an exception for console.log and setTimeout, of course, despite these not being in the language itself, since these are so universal

The amount of packages that qualify for this are vanishingly small.

Though it isn't as pretty as with import maps, one can target node_modules paths directly (for a browser script tag). This works fine for the purposes of a demo when, as is the general situation in which one wishes to run a demo, it is run at the top level of an application and the relative directory structure within node_modules can therefore be deterministically known.

And since projects can make polyglot code with environment-specific code either by inline checks or, as per my preference, separate entry files (e.g., where the Node version supplies a JSDom object as the document object), one can simply import the relevant entry file, so your condition no. 3 is not required. Similarly with condition no. 4. Mitigating the challenges with this are the plethora of libraries previously mentioned like file-fetch, node-fetch, indexeddbshim, etc. which allow for reuse of the same standard APIs across browser and Node as well as browser globals implemented by Node (e.g., URL, URLSearchParams).

The demo application can thus load external third party distribution files out of node_modules (e.g., let's say a jquery distribution and plugins) and any self-contained code within the project it is demoing. The main JavaScript library being imported into the demo should not itself be hard-coding node_modules or it may have problems when imported by other packages, but the HTML file may do so. So your item no. 2 is relevant but only if the library being demoed has external dependencies. In such cases, it is often a large library which needs minification to be practical as a demo anyways which is not my use case. (I'm talking about demoing nicely, modular, single-purpose atomic packages.) But regardless, the file being imported can itself safely include imports of relative paths (e.g., if importing an ESM entry file in the source code where again there are no external dependencies), so condition 1 is not relevant.

So the only relevant condition under this frequent use case is condition 2 (if condition 2 is understood as not importing external dependencies but allowing for project-local relative imports). And hopefully import maps will help even with this condition.

In order to target node_modules, you either need to serve the whole thing (a massive security risk) or run a build process to selectively serve only the files you need.

If you have separate entry files, then you also still need a build process to locate them (outside of node). Note as well that JSDOM tries, but is nowhere near sufficient or equivalent to the actual DOM.

As for node-fetch, the spec for fetch means that a non-browser implementation can never be standards-compliant, as node-fetch is also not.

I think my conditions continue to all apply.

In order to target node_modules, you either need to serve the whole thing (a massive security risk) or run a build process to selectively serve only the files you need.

Not if you're running the demo locally. Again, this is intended especially for rapid development and working on smaller projects.

If you have separate entry files, then you also still need a build process to locate them (outside of node).

Or you can just discover them browsing through the file system or in the project docs.

Note as well that JSDOM tries, but is nowhere near sufficient or equivalent to the actual DOM.

True, but still useful for SSR and such.

As for node-fetch, the spec for fetch means that a non-browser implementation can never be standards-compliant, as node-fetch is also not.

I'm not sure in what regard you're speaking, but even if tweaks are necessary, one can often use an entire API unmodified besides the entry point.

I'm not sure why demos matter; that's not the important audience for software. We should be optimizing for actual production usage, not one-off weekend projects.

I'm not sure why demos matter; that's not the important audience for software. We should be optimizing for actual production usage, not one-off weekend projects.

Many libraries are just developer tools. They don't need a splashy public page. One can smart small, and turn it into a build routine later if desired. I feel we should also be tolerant of different purposes and levels of users in their manner of producing their work.

I personally may add a build routine later solely for copying node_modules files into the repo for use by the likes of GitHub Pages. But my project's own ESM code still doesn't have to go through a build routine in such cases (unless the main library imports other dependencies).