adam-singer

$ kubectl logs scheduler-67845dbc66-n6sf4 |grep ERROR|jq .
Defaulted container "scheduler" out of: scheduler, filter
{
  "timestamp": "2024-09-03T23:19:32.754442Z",
  "level": "ERROR",
  "fields": {
    "message": "Failed to remove_timedout_workers",
    "err": "Error { code: Aborted, messages: [\"Worker ids do not match - Some(002d7a54-48ac-407e-9583-371bf6d81293) != Some(f1cdc851-a5a4-4d38-943b-86b2c7414c1b) for AwaitedAction { version: AwaitedActionVersion(30), action_info: ActionInfo { command_digest: DigestInfo { size_bytes: 3537, hash: \\\"0f9cba98dac0596301da238fdef955942c98fde4c644d7e364934ff8d027144d\\\" }, input_root_digest: DigestInfo { size_bytes: 159, hash: \\\"2c58473407de07d29442edc4ee5b1ed3112efaf45cd1ebe1050106f6b710bc03\\\" }, timeout: 18446744073709551615.999999999s, platform_properties: {\\\"container-image\\\": \\\"docker://ubuntu22:latest\\\", \\\"cpu_count\\\": \\\"50\\\"}, priority: 0, load_timestamp: SystemTime { tv_sec: 0, tv_nsec: 0 }, insert_timestamp: SystemTime { tv_sec: 17254055

adam-singer

  2024-08-29T23:58:08.413385Z ERROR nativelink_util::retry: Not retrying permanent error, attempt: 2, err: Error { code: InvalidArgument, messages: ["status: InvalidArgument, message: \"Tried to send an EOF when pipe is broken : Failed to send EOF in ByteStream::write : In ByteStreamServer::write\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 29 Aug 2024 23:58:07 GMT\", \"content-length\": \"0\"} }", "in GrpcStore::write"] }
  2024-08-29T23:58:08.524986Z ERROR nativelink_worker::local_worker: Error executing action, err: Error { code: InvalidArgument, messages: ["status: InvalidArgument, message: \"Tried to send an EOF when pipe is broken : Failed to send EOF in ByteStream::write : In ByteStreamServer::write\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Thu, 29 Aug 2024 23:58:07 GMT\", \"content-length\": \"0\"} }", "in GrpcStore::write", "in GrpcStore::update()", "Could not upload data to store

adam-singer

{
  "timestamp": "2024-08-29T15:28:05.875122Z",
  "level": "ERROR",
  "fields": {
    "error": "status: Unknown, message: \"status: Unknown, message: \\\"h2 protocol error: error reading a body from connection\\\", details: [], metadata: MetadataMap { headers: {} } : Stream error at byte 78430208 : In ByteStreamServer::write\", details: [], metadata: MetadataMap { headers: {} }"
  },
  "target": "nativelink_service::bytestream_server",
  "span": {
    "request": "Streaming",
    "name": "write"

adam-singer

ERROR: /home/ubuntu/workspace/mongo-1/src/third_party/IntelRDFPMathLib20U1/BUILD.bazel:143:17: Compiling src/third_party/IntelRDFPMathLib20U1/LIBRARY/src/bid32_erfc.c failed: (Exit 34): NOT_FOUND: status: NotFound, message: "Failed to find existing task", details: [], metadata: MetadataMap { headers: {} } : Failed on wait_execution() command
java.io.IOException: io.grpc.StatusRuntimeException: NOT_FOUND: status: NotFound, message: "Failed to find existing task", details: [], metadata: MetadataMap { headers: {} } : Failed on wait_execution() command
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:244)
        at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1505)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:285)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
        at com.google.devtools.build.lib.remot

adam-singer

ERROR: /home/ubuntu/workspace/mongo-1/src/third_party/abseil-cpp/BUILD.bazel:1312:17: Linking src/third_party/abseil-cpp/libabsl_low_level_hash_with_debug.a failed: (Exit 34): 4 errors during bulk transfer:
java.io.IOException: Error while uploading artifact with digest 'c11bd7e038463f0856b976f068d78996dacda8b43eadc7bee4b619856d9ac8bf/260'
java.io.IOException: Error while uploading artifact with digest '4684d1cdcc1abae081927e7fa03a8952b3c4872d368aa1d4f94e7e23d3816487/193'
java.io.IOException: Error while uploading artifact with digest '11765b518d3f2aea019e5480906e096d13ba9910afe495341798e15a4364051a/104'
java.io.IOException: Error while uploading artifact with digest 'ba2216793c7aeed4dd5f8249bedd03ad9aa40b87d4f0059308ce5f97f68cdafe/84'
com.google.devtools.build.lib.remote.common.BulkTransferException: 4 errors during bulk transfer:
java.io.IOException: Error while uploading artifact with digest 'c11bd7e038463f0856b976f068d78996dacda8b43eadc7bee4b619856d9ac8bf/260'
java.io.IOException: Error while uploading ar

adam-singer

ubuntu@ip-172-31-4-32:~/tmp$ ls
app  buck-out  e6b274609d9352a956f81af7504ee0efa04f014a0f93f196cc13d93938706b18.tar.gz  runit.sh
ubuntu@ip-172-31-4-32:~/tmp$ bnash ^C
ubuntu@ip-172-31-4-32:~/tmp$ find .
.
./e6b274609d9352a956f81af7504ee0efa04f014a0f93f196cc13d93938706b18.tar.gz
./.buckconfig
./runit.sh
./app
./app/buck2_data

adam-singer

Action failed: shim//third-party/rust:backtrace-0.3.73 (rustc metadata [pic])
Local command returned non-zero exit code 1
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/shim/7ee685b46ae2051b/rustc/_buck_481ea049fc278ae1' /usr/b ...<omitted>... -party/rust/__backtrace-0.3.73__/rlib-pic-static_pic-metadata-full/backtrace-metadata-full-diag.args (run `buck2 log what-failed` to get the full command)`
stdout:
stderr:
error[E0786]: found invalid metadata files for crate `rustc_demangle`
  --> shim/third-party/rust/backtrace-0.3.73.crate/src/symbolize/mod.rs:13:5
   |
13 | use rustc_demangle::{try_demangle, Demangle};
   |     ^^^^^^^^^^^^^^

adam-singer

$ mkdir -p buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__prelude__/processed/apple/tools/re_compatibility_utils/writable.py/

$ RUST_BACKTRACE=full buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__processor__/processor prelude/apple/tools/re_compatibility_utils/writable.py buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__prelude__/processed/apple/tools/re_compatibility_utils/writable.py/__exec_bit.txt 

$ cat buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__prelude__/processed/apple/tools/re_compatibility_utils/writable.py/__exec_bit.txt
false

adam-singer

Action failed: root//app/buck2_external_cells_bundled:prelude (process_source_file process command_alias.bzl)
Internal error (stage: remote_call_error): Remote Execution Error on execute (GRPC-SESSION-ID): RE: execution with digest a2b21221aa72b67ccc126d38d8c658ac9890080d6b3ee4a6697c1ee92c00fdd8:142: 
Error was returned on the stream by RE: No such file or directory (os error 2) : 
Could not execute command ["buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__processor__/processor", 
"prelude/command_alias.bzl", 
"buck-out/v2/gen/root/904931f735703749/app/buck2_external_cells_bundled/__prelude__/processed/command_alias.bzl/__exec_bit.txt"] : 
--- : Job cancelled because it attempted to execute too many times and failed for operation_id: 
7ea808e2-3f6c-4c04-97db-8b6e5f14b672, maybe_worker_id: Some(4c288cb3-d7dc-4a89-81b2-3102dc77a88c)

adam-singer

2024/04/16 18:17:33 Operating on stack "organization/nativelink/dev". Refreshing...
2024/04/16 18:17:33 Refresh succeeded. Updating stack...
Updating (dev):

 +  pulumi:pulumi:Stack nativelink-dev creating (0s)
 +  kubernetes:yaml:ConfigFile cilium creating (0s)
 +  pulumi:pulumi:Stack nativelink-dev creating (0s) Downloading provider: docker
@ updating....
 +  pulumi:pulumi:Stack nativelink-dev creating (1s) Downloading provider: kubernetes
@ updating....