Troubleshooting GDK

Generic troubleshooting steps

Before attempting the specific troubleshooting steps documented below, try these common troubleshooting steps that resolve most GDK problems:

Run diagnostics: Run gdk doctor to check for common configuration issues. Read and resolve the output.
Update GDK: Run gdk update to ensure you’re on the latest version. Uncommitted changes will be stashed before changing to main branch. All services will be restarted.
Switch branches cleanly: Run gdk switch HEAD to update Node.js modules, Ruby gems, and perform database migrations for your current branch. Many branch-switching issues are resolved this way.
Check service status: Run gdk status to identify any failed services.
Review logs: Run gdk tail <service> or check the log/ directory for error messages from specific services. For advanced log exploration with search and filtering, see GitLab Logs Analysis.
Reset your environment: If you don’t require your database, run gdk pristine to clean up and reset your GDK instance. This command resets databases and cached data.
Use GitLab Duo: Attempt to resolve the issue using Duo for AI-assisted troubleshooting.

If these steps don’t resolve the issue:

Search the documentation for troubleshooting information specific to your issue.
For instance-specific problems, run gdk report to generate a diagnostic report and share the link. See Get help with diagnostic reports.
For general GDK bugs affecting all users, create an issue in the GDK project.
For additional help:
- GitLab team members can use the #gdk channel on the GitLab Slack workspace.
- Community members can use the GitLab community Discord or GitLab Forum.

Get help with diagnostic reports

The gdk report command gathers data from your system, including installed dependencies and their versions, your OS, your architecture, and telemetry data. After gathering this data, your browser automatically opens a new tab where you can paste the report and create a new issue.

To generate a diagnostic report:

Run the following command:
```
gdk report
```
Wait for the command to complete. The command opens your browser automatically.
In the issue description, describe the problem you are facing.
Paste the report inside the issue description. The report should already be in your clipboard.
In the issue title, replace ENTER A TITLE FOR YOUR REPORT with a descriptive title.
Review the contents of your report.
Confirm the issue is marked as confidential.
Submit the issue.
Optional. Use GitLab Duo to help troubleshoot the issue based on the diagnostic report.
Share the issue link when reaching out for help.

Specific problems

GDK troubleshooting information is available for the following:

Installing the GDK
Apple M1/M2 machines
mise
Ruby
Node.js
PostgreSQL
Git
Git packfile accumulation
Webpack
Running tests
Puma
Sidekiq Cluster
Jaeger
Gitaly
Elasticsearch
Homebrew
Live reloading
Praefect
Starting the GDK
Page loading issue due to ad blocker
Stopping and restarting the GDK
Unable to sign in as root
Account verification needed

If you can’t solve your problem, or if you have a problem in another area, open an issue on the GDK issue tracker.

Installing the GDK

`icuc4` setup failed during `make .gitlab-translations` in GDK installation

If you encounter an icuc4 setup error during the make .gitlab-translations step of GDK installation, try the following steps:

Upgrade Xcode to the latest version.
Run brew cleanup to clean up any unnecessary files.
Run brew upgrade to upgrade outdated packages.
Open a new terminal window and continue with the installation of GDK.

PostgreSQL socket creation failed due to GDK directory path being too long

If you encounter a PostgreSQL socket creation failure, it might be because Gitaly is unable to create a Unix socket due to your GDK directory path exceeding the supported character limit.

To fix this issue, try moving the GDK directory to a location with a shorter directory path, then continue with your GDK setup.

Error: Ruby gem extensions fail to compile `unknown type name bool`

You might get an error that states ruby/internal/special_consts.h:137:15: error: unknown type name 'bool'.

This issue occurs when the GCC compiler fails to compile some native Ruby gem extensions (for example, gem io-event). The failure occurs because GCC recently changed the default standard from gnu17 to c23. Ruby built from source by using a GCC version after this change (for example, Fedora 42) is compiled without the HAVE_STDBOOLH config option which is required to compile some extensions.

The workaround is to build Ruby with gnu17 standard before running GDK: CC="gcc -std=gnu17" mise install ruby@3.4.2

Git

The following are possible solutions to problems you might encounter with Git and GDK.

For troubleshooting slow Git operations caused by packfile accumulation, see Git packfile accumulation.

‘Invalid reference name’ when creating a new tag

Make sure that git is configured correctly on your development machine (where GDK runs).

git checkout -b can-I-commit
git commit --allow-empty -m 'I can commit'

`fatal: not a git repository`

If any gdk command gives you the following error:

fatal: not a git repository (or any of the parent directories): .git

Make sure you don’t have gdk aliased in your shell. For example the Git module in prezto has an alias for gdk that lists killed files.

Blobless clones

After you’ve installed GDK, it might have made a blobless partial clone of the gitlab repository. This can cause git blame to be slow or even fail:

$ git blame -- db/structure.sql

remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (1/1), 219.47 KiB | 1.37 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (1/1), 219.32 KiB | 1.39 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (1/1), 219.38 KiB | 1.36 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
...

Here you can see that Git downloads objects one by one.

GDK created a partial clone because it’s a lot faster, but it comes with the downside of not having all objects through history of the repository.

git blame <file>
fatal: You are attempting to fetch XYZ, which is in the commit graph file but not in the object database

To check if your clone is blobless, run:

$ git config remote.origin.partialclonefilter
blob:none

If your clone is a blobless partial clone, this returns blob:none. If you have a blobless clone, you can backfill your repo by running:

# This requires Git version 2.49 or higher.
git backfill

Puma

The following are possible solutions to problems you might encounter with Puma and GDK.

An error has occurred and reported in the system’s low-level handler

If you receive this error message:

An error has occurred and reported in the system's low-level error handler.

This is Puma catching an error that has slipped through from Rails. Here are some suggestions that may assist:

Completely stop your GDK to rule out an out-of-date process running:
```
gdk stop && gdk kill && gdk start
```
Update your GDK as the problem may have already been fixed:
```
gdk update && gdk restart
```
If the problem continues to persist, please raise a GDK Issue ensuring all detail requested in the Issue template is provided.

Puma timeout

Browser shows EOF. Logs show a timeout:

error: GET "/users/sign_in": badgateway: failed after 62s: EOF

Depending on the performance of your development environment, Puma may time out. Increase the timeout as a workaround.

Use environment variables to override the default timeout:

Variable	Type	Description
`GITLAB_RAILS_RACK_TIMEOUT`	integer	Sets `service_timeout`
`GITLAB_RAILS_WAIT_TIMEOUT`	integer	Sets `wait_timeout`

Sidekiq Cluster

GDK uses Sidekiq Cluster (running a single Sidekiq process) by default instead bundle exec sidekiq directly, which is a step towards making development a bit more like production.

Technically, running Sidekiq Cluster with a single Sidekiq process matches the same behavior of running Sidekiq directly, but eventually problems may arise.

If you’re experiencing performance issues or jobs not being picked up, try disabling Sidekiq Cluster by:

Stopping all running processes with gdk stop.
Opening the $GDKROOT/Procfile file.
Removing the SIDEKIQ_WORKERS environment variable from rails-background-jobs.
Starting GDK again with gdk start.

When doing so, please create an issue describing what happened.

Jaeger

If you’re seeing errors such as:

ERROR -- : Failure while sending a batch of spans: Failed to open TCP connection to localhost:14268 (Connection refused - connect(2) for "localhost" port 14268)

This is most likely because Jaeger is not configured in your $GDKROOT/Procfile. The easiest way to fix this is by re-creating your Procfile and then running a gdk reconfigure:

mv Procfile Procfile.old; make Procfile
gdk reconfigure

For more information about Jaeger, visit the distributed tracing GitLab developer documentation.

Gitaly

The following are possible solutions to problems you might encounter with Gitaly and GDK.

`config.toml: no such file or directory`

If you see errors such as:

07:23:16 gitaly.1                | time="2019-05-17T07:23:16-05:00" level=fatal msg="load config" config_path=<path-to-gdk>/gitaly/gitaly.config.toml error="open <path-to-gdk>/gitaly/gitaly.config.toml: no such file or directory"

Somehow, gitaly/gitaly.config.toml is missing. You can re-create this file by running the following in your GDK directory:

make gitaly-setup

Git fails to compile within Gitaly project

If you see the following error when running a gdk update:

ld: library not found for -lgit2

A known fix is to clean your Go cache by running the following from the GDK’s root directory:

go clean -cache
rm -rf gitaly

Now rerun gdk update.

`libegit2.a Error 129`

If you see the following error when running a gdk install or a gdk update:

make[1]: Entering directory '/home/user/gitlab-development-kit/gitaly; error: unknown option `initial-branch=master'
make[1]: *** [Makefile:424: /home/user/gitlab-development-kit/gitaly/_build/deps/libgit2/install/lib/libgit2.a] Error 129

Check which version of Git you’re running with git --version, and compare it against GitLab requirements. You might be running an unsupported version.

If the supported version is not available for you from pre-compiled packages, try following the instructions for:

If that doesn’t give you the supported version, you might have to compile Git from source.

Elasticsearch

Running a spec locally may give you something like the following:

rake aborted!
Gitlab::TaskFailedError: # pkg-config --cflags  -- icu-i18n icu-i18n
Package icu-i18n was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-i18n.pc'
to the PKG_CONFIG_PATH environment variable
No package 'icu-i18n' found
Package icu-i18n was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-i18n.pc'
to the PKG_CONFIG_PATH environment variable
No package 'icu-i18n' found
pkg-config: exit status 1
make: *** [build] Error 2

This indicates that Go is trying to link (unsuccessfully) to brew’s icu4c.

Find the directory where icu-i18n.pc resides:

On macOS, using Homebrew, it is generally in /usr/local/opt/icu4c/lib/pkgconfig or /opt/homebrew/opt/icu4c/lib/pkgconfig
On Ubuntu/Debian it might be in /usr/lib/x86_64-linux-gnu/pkgconfig
On Fedora it is expected to be in /usr/lib64/pkgconfig

You need to add that directory to the PKG_CONFIG_PATH environment variable.

To fix this now, run the following on the command line:

export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH"

To fix this for the future, add the line above to ~/.bash_profile or ~/.zshrc.

Elasticsearch indexer looks for the wrong version of icu4c

You might get the following error when updating the application:

# gitlab.com/gitlab-org/gitlab-elasticsearch-indexer
/usr/local/Cellar/go/1.14.2_1/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: warning: directory not found for option '-L/usr/local/Cellar/icu4c/64.2/lib'
ld: library not found for -licui18n
clang: error: linker command failed with exit code 1 (use -v to see invocation)

make[1]: *** [build] Error 2
make: *** [gitlab-elasticsearch-indexer/bin/gitlab-elasticsearch-indexer] Error 2

This means Go is trying to link to brew’s version of icu4c (64.2 in the example), and failing. This can happen when icu4c is not pinned and got updated. Verify the version with:

$ ls /usr/local/Cellar/icu4c
66.1

Clean Go’s cache to fix this error. From the GDK’s root directory:

cd gitlab-elasticsearch-indexer/
go clean -cache

Homebrew

Most brew problems can be figured out by running:

brew doctor

However, older installations may have significant cruft leftover from previous installations and updates. To manually remove outdated downloads for all formulae, casks, and stale lock files, run:

brew cleanup

For more information on uninstalling old versions of a formula, see the Homebrew FAQ. For additional troubleshooting information, see the Homebrew Common Issues page.

Live reloading

If you previously compiled production assets with bundle exec rake gitlab:assets:compile, the GDK serves the assets from the public/assets/ directory, which means that changing SCSS files doesn’t have any effect in development until you recompile the assets manually.

To re-enable live reloading of CSS in development, remove the public/assets/ directory and restart GDK.

Praefect

get shard for “default”: primary is not healthy

From the GDK’s root directory:

cd gitaly/ruby
bundle install

You may need to run a gdk restart for the changes to take effect.

`/home/user/gitlab-development-kit/gitaly/_build/bin/praefect`: No such file or directory

You might encounter the following error while running Gitaly database migrations:

support/migrate-praefect: line 4: /home/user/gitlab-development-kit/gitaly/_build/bin/praefect: No such file or directory
migrate failed
make: *** [_postgresql-seed-praefect] Error 1

This means /gitaly/_build/bin/praefect is missing. To re-create this executable file, run the following in your GDK directory:

make gitaly-update

Updating the GDK

Run `gdk doctor`

As a general rule, if you encounter errors when you run gdk update, you should run gdk doctor and follow the suggestions it returns. They might resolve your issue.

You may use gdk doctor --correct to autocorrect trivial issues.

Blank page after update

If you encounter a blank page after performing an update, this is because of a hanging process. To solve the issue, follow these steps:

Identify the hanging process ID (PID) bound to port 3000:
```
lsof -i tcp:3000
```
Using the returned PID, kill the hanging process:
```
kill -9 <process id>
```
Restart GDK services:
```
gdk restart
```

If you’ve changed your default port, use the following command to find your custom port number: gdk config get port

Error due to `Net::OpenTimeout: Failed to open TCP connection to rubygems.org:443`

When you run gdk update you might get an error like the following:

ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)

This indicates that bundle failed to connect to the rubygems.org server.

If you are connected to the network and other network activities are working (i.e. ping gitlab.com), then this normally indicates an outage of rubygems.org. You can try manually running bundle update in the GDK root folder, and if it fails with a similar network error, you know this is the cause.

However, in some cases, even though bundle update is otherwise working successfully, you might get an error like the following:

INFO: Installing gitlab-development-kit Ruby gem..
ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
    Net::OpenTimeout: Failed to open TCP connection to rubygems.org:443 (execution expired) (https://rubygems.org/specs.4.8.gz)

This can happen if the IPv6 access to rubygems.org is having an outage, but IPv4 access is still working.

Using this comment as an example, you can add :ipv4_fallback_enabled: true to your ~/.gemrc to work around this until this rubygems pull request gets merged.

If that doens’t work for some reason, you can alternately go into your operating system network settings and disable IPv6 for your network adapter. Refer to your operating system documentation for detailed instructions.

Starting the GDK

Unable to start as CE (FOSS_ONLY)

If your GDK enters an infinite loop when running FOSS_ONLY=1 gdk start, try to remove - { name: 'group_saml' } option from your config/gitlab.yml. The favicon should be blue when in CE and green when in EE.

Error due to `ActionController::InvalidAuthenticityToken`

If you encounter an ActionController::InvalidAuthenticityToken error when starting GDK, try the following steps:

Hard refresh your browser to clear the cache. For more information, see How to hard refresh your browser.
Stop all GDK processes using gdk kill, then restart GDK with gdk start.
Delete all browser cache, cookies, local storage, and other related data for the relevant hostname.

Error starting `timeout: down: /Users/foo/gdk/services/rails-background-jobs: 0s, want up`

Check if a background job process is unexpectedly running:

Search for processes related to background jobs: ps aux | grep rails-background-jobs
If runit is currently trying to start rails-background-jobs then you may find: runsvdir -P /Users/foo/gdk/services log: atal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012runsv rails-background-jobs: fatal: unable to lock supervise/lock: temporary failure\012
Stop GDK services: gdk stop
If ps aux | grep rails-background-jobs shows that runsv rails-background-jobs is still running, then it is preventing rails-background-jobs from starting
Kill the process by following Stopping and restarting the GDK

If the above is not the problem, then confirm you can run rails-background-jobs manually: RAILS_ENV=development ./bin/background_jobs start_foreground.

Error `Listen::Error::INotifyMaxWatchesExceeded`

Depending on your system configuration, you can receive errors about an excess or maximum number of inotify watchers. This error can appear when running gdk tail.

To solve this problem, increase the maximum allowed inotify watchers. For more information, see relevant documentation for the listen gem.

Page loading issue due to ad blocker

Some ad blockers, such as uBlock Origin, might prevent web-vitals.js from loading properly. This results in errors such as:

Failed to load 'pages.projects.jobs.index.js' entrypoint

To resolve this issue, disable your ad blocker and reload the application.

Stopping and restarting the GDK

Sometimes the GDK will fail to stop or restart. This is sometimes caused by processes not shutting down gracefully and can prevent subsequent attempts to stop/start the GDK.

You might see something like the following after running gdk stop:

kill: run: ./services/rails-background-jobs: (pid 89668) 98s, normally down, want down

To kill off the rogue processes, run gdk kill.

Unable to log in as root

If all the services are running after you run gdk install, but you cannot log in as root, you can reset the password through the Rails console.

For more information about the Ruby on Rails console in GitLab, see Rails console.

Open a new Rails console:

cd <gdk-dir>/gitlab
bundle exec rails console

In the Rails console, update the password for the root user:

user = User.find_by_username('root')
user.update!(password: 'newpassword')

To find the user IDs for all current users, run:

User.all

Account verification needed

Sometimes, after resetting data, or on initial setup, you may see a message like this when logging in for the first time:

Before you can run pipelines, we need to verify your account.
We won't ask you for this information again. It will never be used for marketing purposes.

This is often caused by having GITLAB_SIMULATE_SAAS=1 in your env.runit file. You can set it to 0 if it’s not needed and then restart your GDK. But if you do need it to be set, such as for running the GitLab AI Gateway, you can try this from the Rails console:

Option 1: Create validation records

root = User.first
Users::CreditCardValidation.create!(user: root, credit_card_validated_at: Time.zone.now)
Users::PhoneNumberValidation.create!(user: root, validated_at: Time.zone.now, country: "US", phone_number: "7665554269", international_dial_code: "+1")

Option 2: Add identity verification exemption

root = User.first
root.add_identity_verification_exemption("I'm local")

Orphaned `workerd` processes

Sometimes, workerd hangs and prevents GDK from working. You can try and kill all workerd processes:

killall -9 workerd

If that doesn’t help, you can try disabling gitlab-http-router.

Last updated on March 25, 2026