Irregular segmentation faults when establishing connections

[ThomasCrambert]

Hello 👋,

I've been trying to troubleshoot segmentation faults that we are getting on our production environment.
They seem to appear when ActiveRecord attempts to reestablish a connection, most of them were from the processing of ActiveRecord's connection pool reaper.

As the issue comes from the native extension and is very low level, it made more sense to me to open an issue here to ask for guidance.

Here is the relevant part of the crash reports:

/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb:167: [BUG] Segmentation fault at 0x0000000000000000

-- Ruby level backtrace information ----------------------------------------
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:94:in 'block in spawn_thread'
<internal:array>:228:in 'each'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:95:in 'block (2 levels) in spawn_thread'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:578:in 'reaper_lock'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:578:in 'synchronize'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool/reaper.rb:101:in 'block (3 levels) in spawn_thread'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:811:in 'preconnect'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:979:in 'sequential_maintenance'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:967:in 'block in sequential_maintenance'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:812:in 'block in preconnect'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:839:in 'connect!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:817:in 'verify!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activesupport/lib/active_support/concurrency/null_lock.rb:9:in 'synchronize'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:830:in 'block in verify!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:715:in 'reconnect!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activesupport/lib/active_support/concurrency/null_lock.rb:9:in 'synchronize'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:716:in 'block in reconnect!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:1290:in 'attempt_configure_connection'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:727:in 'block (2 levels) in reconnect!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:410:in 'reset_transaction'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:729:in 'block (3 levels) in reconnect!'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:1017:in 'configure_connection'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb:310:in 'schema_search_path'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:109:in 'query_value'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb:15:in 'query'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:613:in 'internal_execute'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:569:in 'raw_execute'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:1206:in 'log'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activesupport/lib/active_support/notifications/instrumenter.rb:58:in 'instrument'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:570:in 'block in raw_execute'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:1055:in 'with_raw_connection'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activesupport/lib/active_support/concurrency/null_lock.rb:9:in 'synchronize'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:1086:in 'block in with_raw_connection'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:571:in 'block (2 levels) in raw_execute'
/usr/local/bundle/bundler/gems/rails-2ac86a8a9e32/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb:167:in 'perform_query'

-- C level backtrace information -------------------------------------------
/usr/local/lib/libruby.so.3.4(rb_print_backtrace+0x8) [0x7f41a69212da] /usr/src/ruby/vm_dump.c:823
/usr/local/lib/libruby.so.3.4(rb_vm_bugreport) /usr/src/ruby/vm_dump.c:1155
/usr/local/lib/libruby.so.3.4(rb_bug_for_fatal_signal+0x100) [0x7f41a67070b0] /usr/src/ruby/error.c:1130
/usr/local/lib/libruby.so.3.4(sigsegv+0x42) [0x7f41a68639b2] /usr/src/ruby/signal.c:935
/lib/x86_64-linux-gnu/libc.so.6(0x7f41a5f80050) [0x7f41a5f80050]
/usr/lib/x86_64-linux-gnu/libpq.so.5(dopr.constprop.0+0xd48) [0x7f417781e958] ./build/../src/port/snprintf.c:1459
/usr/lib/x86_64-linux-gnu/libpq.so.5(pg_vsnprintf+0x57) [0x7f417781eea7] ./build/../src/port/snprintf.c:195
/usr/lib/x86_64-linux-gnu/libpq.so.5(appendPQExpBufferVA+0x32) [0x7f417780eee2] ./build/../src/interfaces/libpq/pqexpbuffer.c:308
/usr/lib/x86_64-linux-gnu/libpq.so.5(libpq_append_conn_error+0xe7) [0x7f4177806977] ./build/../src/interfaces/libpq/fe-misc.c:1420
/usr/lib/x86_64-linux-gnu/libpq.so.5(pqParseInput3+0xc5) [0x7f417780a5d5] ./build/../src/interfaces/libpq/fe-protocol3.c:504
/usr/lib/x86_64-linux-gnu/libpq.so.5(PQgetResult+0x18) [0x7f4177802fd8] ./build/../src/interfaces/libpq/fe-exec.c:2038
/usr/local/bundle/gems/pg-1.6.3/lib/pg_ext.so(gvl_PQgetResult_skeleton+0x23) [0x7f4177853c7f] /usr/local/bundle/gems/pg-1.6.3/ext/gvl_wrappers.c:25
/usr/local/lib/libruby.so.3.4(rb_nogvl+0x29f) [0x7f41a68b4d3f] /usr/src/ruby/thread.c:1576
/usr/local/bundle/gems/pg-1.6.3/lib/pg_ext.so(gvl_PQgetResult+0x3e) [0x7f41778546df] /usr/local/bundle/gems/pg-1.6.3/ext/gvl_wrappers.c:27
/usr/local/bundle/gems/pg-1.6.3/lib/pg_ext.so(pgconn_async_get_last_result+0x58) [0x7f417786134b] /usr/local/bundle/gems/pg-1.6.3/ext/pg_connection.c:3272
/usr/local/bundle/gems/pg-1.6.3/lib/pg_ext.so(pgconn_async_exec+0x48) [0x7f4177861615] /usr/local/bundle/gems/pg-1.6.3/ext/pg_connection.c:3419
/usr/local/lib/libruby.so.3.4(vm_call_cfunc_with_frame_+0x115) [0x7f41a68ed675] /usr/src/ruby/vm_insnhelper.c:3794
/usr/local/lib/libruby.so.3.4(rb_vm_invokesuper+0x102) [0x7f41a691a212] /usr/src/ruby/vm_insnhelper.c:5965

I tried to dive into both activerecord and ruby-pg codebases to come up with a reproduction script, but I couldn't manage to have a scenario creating a segmentation fault, all my attempts resulted in normal errors.

The technical stack is the following:

• Ruby 3.4.8
• Rails 8.1
• ruby-pg 1.6.3
• Pitchfork as a web server, SolidQueue as a job orchestrator (issue is happening under both contexts)
• AWS RDS as a postgres backend (running with Postgresql 15)
• PgBouncer in between in transaction mode

Do you have any idea how could make it end up like that?

As it is still happening frequently, we can add additional monitoring to investigate any lead you may have.

Please let me know if you require additional details!

[jtarchie]

I think I have a minimal reproducible example. It isn't the exact same code path, but we experienced something similar with pgsync on Mac, which made me hunt this down.

require "pg"

# Any remote PostgreSQL URL using SSL
ssl_url = ENV.fetch("DATABASE_URL") # e.g. postgres://user:pass@remote-host:5432/db?sslmode=require

# Step 1: Parent opens SSL connection
parent_conn = PG.connect(ssl_url)
parent_conn.exec("SELECT 1")

# Step 2: Fork
pid = fork do
  # Step 3: Child opens new SSL connection → segfault
  child_conn = PG.connect(ssl_url)
  child_conn.exec("SELECT 1")
  child_conn.close
end

Process.wait(pid)
puts "Child exit: #{$?.exitstatus}" # nil (crashed)
parent_conn.close

Versions

$ ruby -v
ruby 4.0.1 (2026-01-13 revision e04267a14b) +PRISM [arm64-darwin25]
$ gem list pg
pg (1.6.3 arm64-darwin)

Totally had LLM explain all the things:

On Ruby 4.0.1 (arm64-darwin25), forking after a parent process has established an SSL connection to a remote PostgreSQL server causes the child process to segfault when it attempts to open its own new SSL connection. The crash occurs inside _os_log_preferences_refresh / _rb_thread_prevent_fork in libruby.4.0.dylib, and appears to be caused by inherited OpenSSL/libssl state that is not fork-safe. This affects tools like pgsync that open a connection for metadata queries, then fork workers to sync individual tables in parallel.

I want to reference a similar issue: grosser/parallel#337

[larskanis]

The reproducer doesn't segfault on Linux for me. Both processes work just fine.

$ ruby -v
ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [aarch64-linux]
$ gem list pg
pg (1.6.3 aarch64-linux)
$ DATABASE_URL="host=localhost port=54321 dbname=postgres" ruby -Ilib test-segfault-sprintf.rb 
{"library" => "OpenSSL", "key_bits" => "256", "cipher" => "TLS_AES_256_GCM_SHA384", "compression" => "off", "protocol" => "TLSv1.3", "alpn" => "postgresql"}
#<PG::Result:0x0000ffffa1276680 status=PGRES_TUPLES_OK ntuples=1 nfields=1 cmd_tuples=1>
Child exit: 0

With the following script:

require "pg"

# Any remote PostgreSQL URL using SSL
ssl_url = ENV.fetch("DATABASE_URL") # e.g. postgres://user:pass@remote-host:5432/db?sslmode=require

# Step 1: Parent opens SSL connection
parent_conn = PG.connect(ssl_url)
parent_conn.exec("SELECT 1")
p parent_conn.ssl_attributes

# Step 2: Fork
pid = fork do
  # Step 3: Child opens new SSL connection → segfault
  child_conn = PG.connect(ssl_url)
  p child_conn.exec("SELECT 1")
  child_conn.close
end

Process.wait(pid)
puts "Child exit: #{$?.exitstatus}" # nil (crashed)
parent_conn.close

Could you please post the ruby crash report to it? It might be the longstanding issue with kerberos on Macos #663 .

[larskanis]

/usr/lib/x86_64-linux-gnu/libpq.so.5(pqParseInput3+0xc5) [0x7f417780a5d5] ./build/../src/interfaces/libpq/fe-protocol3.c:504

Your Linux stacktrace crashes in handleSyncLoss. This is a sign of a broken bytestream in the communication, possibly caused by OpenSSL. No idea why the sprintf call crashes as a result of broken synchronization, so that you don't get a proper error message from the connection attempt. Maybe something wrong with gettext.

[jtarchie]

Yep, it is totally that other issue #663