Fix Sonatype Nexus OrientDB

gistfile1.txt

$ ls /opt/sonatype/sonatype-work/nexus3/db/
OSystem           accesslog         analytics         audit             component         config            model.properties  security

$ docker-compose exec nexus java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component admin admin
REBUILD INDEX *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
DISCONNECT

I was trying to execute REBUILD INDEX * and got error:

Rebuilding index(es)...$ANSI{green {db=component}} Error during index rebuild
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
        DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
        at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2498)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
        at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:940)
        at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1040)
        at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:539)
        at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:500)
        at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:403)
        at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.rebuild(OIndexAbstractDelegate.java:167)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLRebuildIndex.execute(OCommandExecutorSQLRebuildIndex.java:91)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.sqlCommand(OConsoleDatabaseApp.java:3076)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.rebuildIndex(OConsoleDatabaseApp.java:1273)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
        at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
        at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
        at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
        DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671

Please advise.

any update?

I get this:

Error creating history file java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at com.orientechnologies.common.console.TTYConsoleReader.getHistoryFile(TTYConsoleReader.java:554) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:89) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) History file not found java.io.FileNotFoundException: .orientdb_history (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileReader.<init>(FileReader.java:72) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:92) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) Exception in thread "main" com.orientechnologies.common.exception.OSystemException: Cannot access to the input stream. Check permissions of running process at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:112) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

any update on this @bigmyx ?

I get this:

Error creating history file java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at com.orientechnologies.common.console.TTYConsoleReader.getHistoryFile(TTYConsoleReader.java:554) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:89) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) History file not found java.io.FileNotFoundException: .orientdb_history (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileReader.<init>(FileReader.java:72) at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:92) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) Exception in thread "main" com.orientechnologies.common.exception.OSystemException: Cannot access to the input stream. Check permissions of running process at com.orientechnologies.common.console.TTYConsoleReader.<init>(TTYConsoleReader.java:112) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:143) at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

In case anyone else encounters this issue, try running the $ docker-compose exec nexus java -jar ./lib/support/nexus-orient-console.jar command with the root user specified:

$ docker-compose exec -u root nexus java -jar ./lib/support/nexus-orient-console.jar

After running the commands from the original snippet, make sure to restore ownership of all database files back to the nexus user and nexus group, as the modified files will now belong to the root user!

If you are getting bigmyx's error:

Rebuilding index(es)...$ANSI{green {db=component}} Error during index rebuild
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #42:150596: found duplicated key 'OCompositeKey{keys=[#17:3, null, v2/-/blobs/sha256:f2997afb97d16b6fbbbd9315ee03c6d4348af4edcec7aa31831b1093986792ff]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #42:107671
DB name="component" INDEX=asset_bucket_component_name_idx RID=#42:107671
at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2498)
at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:940)
at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1040)
at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:539)
at com.orientechnologies.orient.core.index.OIndexAbstract.rebuild(OIndexAbstract.java:500)
at com.orientechnologies.orient.core.index.OIndexAbstra

You can truncate the record as described here: https://mariosk.wordpress.com/2019/09/07/how-a-nexus-repository-manager-corruption-led-to-a-mini-odyssey/
After i fixed all the dups to fix index: asset_bucket_component_name_idx i encountered a lot more appeared in the browse_node_repository_name_parent_path_name_idx index, after fixing a couple of dozes manually i dropped the class browse_node as mentioned here: https://issues.sonatype.org/browse/NEXUS-16419 this should only have affect on the webui and after restart you the indexes are repopulated.

Another note about this: With nexus OSS 3.15.1, when it ran out of disk space, it created WAL files with zero file size. These caused orientdb to croak with null pointer exceptions on startup. I just had to remove the zero size files, and then the above commands worked perfectly.

Many thanks

~/go/src/etcd-io/etcd/tools/etcd-dump-db (main ✔) docker exec -u 0 -ti af9b0952e351 bash
                              
[root@af9b0952e351 /]# java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar 
                                                                      
OrientDB console v.2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x) https://www.orientdb.com
Type 'help' to display all the supported commands.    
orientdb> CONNECT PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component admin admin
                                   
Connecting to database [PLOCAL:/opt/sonatype/sonatype-work/nexus3/db/component] with user 'admin'...
2023-08-21 08:11:10:770 WARNI {db=component} Storage 'component' was not closed properly. Will try to recover from write ahead log... [OLocalPaginatedStorage]                                                     
2023-08-21 08:11:10:779 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRec
ord{lsn=LSN{segment=23, position=160518}} will be skipped during data restore [OLocalPaginatedStorage]
2023-08-21 08:11:10:779 WARNI {db=component} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=23, position=160525}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null,previousCheckpoint=LSN{segment=23, position=160478}} will be skipped during data restore [OLocalPaginatedStorage]
2023-08-21 08:11:10:780 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=23, position=160565}} will be skipped during data restore [OLocalPaginatedStorage]OK
orientdb {db=component}> REBUILD INDEX *        
                                   

Rebuilding index(es)...
Rebuilt index(es). Found 108 link(s) in 10.375000 sec(s).

Index(es) rebuilt successfully
orientdb {db=component}> REPAIR DATABASE --fix-graph

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> REPAIR DATABASE --fix-links

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> 
orientdb {db=component}> REPAIR DATABASE --fix-ridbags

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> REPAIR DATABASE --fix-bonsai 

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors) 
orientdb {db=component}> disconnect  

Disconnecting from the database [component]...OK
orientdb>