fast-tags vs hasktags vs ghc-tags

0readme.md

⚠️ Testing Emacs tags generation specifically. I've updated one which fast-tags was lacking -e.

I thought I'd upgrade my hasktags to fast-tags, but discovered that it's not much faster. The much bigger saving is to pipe fd into it rather than letting either fast-tags/hasktags do its own recursive directory searching. They appear to be egregiously slow in comparison to fd.

I added ghc-tags, and that's the slowest for recursive cold run. It's the fastest on a warm run with fd.

Confirmation that they're all producing outputs of roughly similar size.

bash-3.2$ ls -alh .*tags
-rw-r--r-- 1 chris staff 1.2M Oct 29 15:44 .fast-tags
-rw-r--r-- 1 chris staff 1.7M Oct 29 15:52 .ghc-tags
-rw-r--r-- 1 chris staff 1.1M Oct 29 15:51 .tags

fast-tags-hasktags.txt

# Using the recursive directory checker it's not any faster.

bash-3.2$ hasktags . -o .tags +RTS -s
   4,280,500,176 bytes allocated in the heap
     938,002,744 bytes copied during GC
     205,046,592 bytes maximum residency (9 sample(s))
         552,128 bytes maximum slop
             406 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1018 colls,     0 par    0.221s   0.224s     0.0002s    0.0010s
  Gen  1         9 colls,     0 par    0.132s   0.181s     0.0201s    0.0818s

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    0.914s  (  2.410s elapsed)
  GC      time    0.353s  (  0.404s elapsed)
  EXIT    time    0.000s  (  0.010s elapsed)
  Total   time    1.267s  (  2.826s elapsed)

  %GC     time       0.0%  (0.0% elapsed)

  Alloc rate    4,683,532,169 bytes per MUT second

  Productivity  72.1% of total user, 85.3% of total elapsed

bash-3.2$ fast-tags -R . -e -o .fast-tags +RTS -s -RTS 
   3,747,060,616 bytes allocated in the heap
     918,907,888 bytes copied during GC
     143,344,664 bytes maximum residency (13 sample(s))
         637,928 bytes maximum slop
             287 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       852 colls,     0 par    0.257s   0.266s     0.0003s    0.0014s
  Gen  1        13 colls,     0 par    0.125s   0.157s     0.0121s    0.0638s

  TASKS: 19 (1 bound, 13 peak workers (18 total), using -N1)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    1.005s  (  4.887s elapsed)
  GC      time    0.382s  (  0.423s elapsed)
  EXIT    time    0.000s  (  0.004s elapsed)
  Total   time    1.388s  (  5.315s elapsed)

  Alloc rate    3,726,594,160 bytes per MUT second

  Productivity  72.5% of total user, 91.9% of total elapsed
  
  bash-3.2$ ghc-tags . -e -o .ghc-tags +RTS -s
  16,156,718,048 bytes allocated in the heap
     956,515,864 bytes copied during GC
      35,579,832 bytes maximum residency (20 sample(s))
       1,601,696 bytes maximum slop
             100 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      2956 colls,  2918 par    0.700s   0.443s     0.0001s    0.0019s
  Gen  1        20 colls,    17 par    0.303s   0.179s     0.0090s    0.0404s

  Parallel GC work balance: 48.37% (serial 0%, perfect 100%)

  TASKS: 21 (1 bound, 20 peak workers (20 total), using -N5)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    3.252s  (  8.332s elapsed)
  GC      time    1.003s  (  0.623s elapsed)
  EXIT    time    0.000s  (  0.006s elapsed)
  Total   time    4.256s  (  8.962s elapsed)

  Alloc rate    4,967,580,582 bytes per MUT second

  Productivity  76.4% of total user, 93.0% of total elapsed

using-fd-is-faster.txt

# Using fd, it's twice as fast. Both both are 1s faster.

bash-3.2$ fd '\.hs$' | xargs hasktags -o .tags +RTS -s -RTS
   3,127,195,584 bytes allocated in the heap
     179,081,400 bytes copied during GC
      44,187,464 bytes maximum residency (7 sample(s))
       2,310,512 bytes maximum slop
             104 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       743 colls,     0 par    0.066s   0.068s     0.0001s    0.0005s
  Gen  1         7 colls,     0 par    0.032s   0.040s     0.0058s    0.0204s

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    0.466s  (  0.494s elapsed)
  GC      time    0.098s  (  0.108s elapsed)
  EXIT    time    0.000s  (  0.008s elapsed)
  Total   time    0.564s  (  0.611s elapsed)

  %GC     time       0.0%  (0.0% elapsed)

  Alloc rate    6,711,843,578 bytes per MUT second

  Productivity  82.6% of total user, 80.8% of total elapsed

bash-3.2$ fd '\.hs$' | xargs fast-tags -e -o .fast-tags +RTS -s -RTS
   1,960,643,664 bytes allocated in the heap
     230,543,136 bytes copied during GC
      38,892,520 bytes maximum residency (13 sample(s))
         794,440 bytes maximum slop
              68 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       424 colls,     0 par    0.076s   0.079s     0.0002s    0.0013s
  Gen  1        13 colls,     0 par    0.028s   0.032s     0.0024s    0.0056s

  TASKS: 7 (1 bound, 6 peak workers (6 total), using -N1)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    0.522s  (  0.548s elapsed)
  GC      time    0.104s  (  0.111s elapsed)
  EXIT    time    0.000s  (  0.003s elapsed)
  Total   time    0.626s  (  0.664s elapsed)

  Alloc rate    3,756,159,063 bytes per MUT second

  Productivity  83.4% of total user, 82.6% of total elapsed
  
  bash-3.2$ fd '\.hs$' | xargs ghc-tags -e -o .ghc-tags +RTS -s -RTS
   5,002,107,080 bytes allocated in the heap
     819,688,848 bytes copied during GC
      36,877,696 bytes maximum residency (17 sample(s))
       1,804,928 bytes maximum slop
              99 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       364 colls,   320 par    0.567s   0.283s     0.0008s    0.0019s
  Gen  1        17 colls,    13 par    0.268s   0.130s     0.0076s    0.0143s

  Parallel GC work balance: 59.33% (serial 0%, perfect 100%)

  TASKS: 18 (1 bound, 17 peak workers (17 total), using -N5)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    1.005s  (  0.334s elapsed)
  GC      time    0.835s  (  0.413s elapsed)
  EXIT    time    0.000s  (  0.007s elapsed)
  Total   time    1.841s  (  0.755s elapsed)

  Alloc rate    4,977,027,836 bytes per MUT second

  Productivity  54.6% of total user, 44.3% of total elapsed
  
  
 # Subsequent runs are faster, though
 
 bash-3.2$ fd '\.hs$' | xargs ghc-tags -e -o .ghc-tags +RTS -s -RTS
     296,040,760 bytes allocated in the heap
      55,523,128 bytes copied during GC
      13,550,696 bytes maximum residency (5 sample(s))
         813,976 bytes maximum slop
              52 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0        55 colls,    27 par    0.020s   0.015s     0.0003s    0.0015s
  Gen  1         5 colls,     1 par    0.035s   0.030s     0.0061s    0.0083s

  Parallel GC work balance: 61.94% (serial 0%, perfect 100%)

  TASKS: 16 (1 bound, 15 peak workers (15 total), using -N5)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    0.066s  (  0.067s elapsed)
  GC      time    0.055s  (  0.045s elapsed)
  EXIT    time    0.000s  (  0.011s elapsed)
  Total   time    0.121s  (  0.124s elapsed)

  Alloc rate    4,455,693,923 bytes per MUT second

  Productivity  54.7% of total user, 54.0% of total elapsed
  
 # But only with fd, without even subsequent runs are very slow
 
   11,453,153,576 bytes allocated in the heap
     217,803,144 bytes copied during GC
      19,933,656 bytes maximum residency (8 sample(s))
       1,394,216 bytes maximum slop
              72 MiB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      2605 colls,  2585 par    0.165s   0.163s     0.0001s    0.0018s
  Gen  1         8 colls,     4 par    0.082s   0.055s     0.0069s    0.0109s

  Parallel GC work balance: 27.54% (serial 0%, perfect 100%)

  TASKS: 20 (1 bound, 19 peak workers (19 total), using -N5)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.000s  (  0.001s elapsed)
  MUT     time    2.250s  (  9.020s elapsed)
  GC      time    0.247s  (  0.218s elapsed)
  EXIT    time    0.000s  (  0.004s elapsed)
  Total   time    2.498s  (  9.243s elapsed)

  Alloc rate    5,089,360,821 bytes per MUT second

  Productivity  90.1% of total user, 97.6% of total elapsed