`git push` from command line fails — Sztabina Smart HTTP handler returns empty response (SZ-140)
rk@tigase.net opened 4 weeks ago

Description

Pushing to a Sztab git repository from the command line fails with:

XML error: no element found
error: no DAV locking support on http://git.sztab.local/git/<repo>.git/
fatal: git-http-push failed

Component

Sztabina

Root Cause

Sztabina's info/refs?service=git-receive-pack endpoint returns HTTP 200 with an empty body instead of the proper Smart HTTP pktline advertisement. Git interprets the empty response as Smart HTTP being unavailable and falls back to WebDAV push, which Sztabina does not support.

Impact

Developers cannot push to Sztab git repositories from the command line. UI-based workflows (branch creation, PR merge) are unaffected.

Workaround

None currently. Use Sztab UI for all git operations.

Steps to Reproduce

git clone http://git.sztab.local/git/<repo>.git
# make a commit
git push
  • rk@tigase.net commented 4 weeks ago
    rksuma@Ramakrishnans-MacBook-Pro sztab % git checkout -b bugfix/SZ-140-git-push-command-fails
    Switched to a new branch 'bugfix/SZ-140-git-push-command-fails'
    rksuma@Ramakrishnans-MacBook-Pro sztab % 
    
    
  • rk@tigase.net changed state to 'In Progress' 4 weeks ago
    Previous Value Current Value
    Open
    In Progress
  • rk@tigase.net commented 4 weeks ago

    I found that the root cause is the missing CGI environment variables required for git-http-backend (the CGI binary that handles Smart HTTP push/fetch) to function properly.

    git-http-backend requires three CGI environment variables to function correctly for push:

    • QUERY_STRING — carries service=git-receive-pack, telling the backend which protocol to advertise on info/refs
    • REQUEST_METHOD — needed for POST dispatch on the actual git-receive-pack call
    • CONTENT_TYPE — needed to parse the POST body during push

    Go's cgi.Handler auto-derives many CGI vars from the incoming request, but when you provide an explicit Env slice it can't reliably inject all three of the above. They were absent from the Env slice in GitHTTPHandler.ServeHTTP.

    Without QUERY_STRING, git-http-backend sees no service parameter on info/refs, produces no output, and returns an empty 200 body. Git interprets an empty Smart HTTP response as Smart HTTP being unavailable and falls back to legacy WebDAV push — which Sztabina doesn't implement — producing the no DAV locking support error.

    Fix: explicitly add QUERY_STRING, REQUEST_METHOD, and CONTENT_TYPE to the Env slice passed to cgi.Handler.

  • rk@tigase.net commented 4 weeks ago

    No, the fix above did not work, although that change was warranted. Still debugging...

  • rk@tigase.net commented 4 weeks ago

    I wasted 2 hours. I don't have a root cause - analysis in https://tigase.dev/sztab/~issues/140#IssueComment-131417 is wrong.

    Caddy logs show that git push is not even hitting caddy!

  • rk@tigase.net commented 3 weeks ago

    FIXED!!!!!

    rksuma@Ramakrishnans-MacBook-Pro genai-project % git push                                                       
    Enumerating objects: 3, done.
    Counting objects: 100% (3/3), done.
    Delta compression using up to 12 threads
    Compressing objects: 100% (2/2), done.
    Writing objects: 100% (3/3), 600 bytes | 600.00 KiB/s, done.
    Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
    remote: 2026/04/06 06:08:56 notify: forwarded repo=genai-project ref=refs/heads/feature/llm-deployment commits=1
    To http://git.sztab.local/git/genai-project.git
     * [new branch]      feature/llm-deployment -> feature/llm-deployment
    branch 'feature/llm-deployment' set up to track 'origin/feature/llm-deployment'.
    rksuma@Ramakrishnans-MacBook-Pro genai-project % 
    
    

    True root cause of SZ-140: Caddy's site block was configured for localhost only. Git sends Host: git.sztab.local, which didn't match the site block, so Caddy logged every git request as NOP and returned an empty 200. The fix was adding http://git.sztab.local as a second address on the site block, plus the global options block {} to satisfy Caddy's multi-address syntax requirement, plus changing respond @ai_bots 403 to handle @ai_bots { respond 403 }.

  • rk@tigase.net commented 3 weeks ago

    SZ-140 Post-Mortem — git push Command-Line Failure

    Primary Root Cause Caddy's site block was configured with a single address resolving to localhost. Git clients send Host: git.sztab.local, which did not match the site block. Caddy logged every git request as NOP — no handler matched, no routing occurred — and returned an empty 200 OK. Git interpreted the empty Smart HTTP response as the server not supporting Smart HTTP and fell back to legacy WebDAV push, which Sztabina does not implement, producing the no DAV locking support error.

    Fix: Added http://git.sztab.local as a second address on the Caddy site block, added a required global options block {} for multi-address syntax, and changed respond

    @ai_bots 403 to handle @ai_bots { respond 403 } 
    

    to satisfy the Caddy parser.

    Secondary Fix — Sztabina CGI Environment git-http-backend requires QUERY_STRING, REQUEST_METHOD, and CONTENT_TYPE to be explicitly present in the CGI environment. These were absent from the Env slice passed to Go's cgi.Handler. Without QUERY_STRING, git-http-backend cannot read the service=git-receive-pack parameter on info/refs and produces no output. This would have caused push failures for authenticated requests even after the Caddy fix. Secondary Fix — Caddy WWW-Authenticate Propagation

    Added copy_headers WWW-Authenticate to the forward_auth block so that 401 challenges from the auth validator are forwarded to git clients, enabling credential prompting and retry rather than silent failure.

    Why It Wasn't Caught in 1.10.0 Testing Release testing used a cached PAT in the macOS keychain, which git sent automatically on every request. The primary Caddy routing bug was masked because the Host header happened to match during that test environment. CLI push was not covered in regression_pr.sh.

    Follow-up Add a git push smoke test to regression_pr.sh covering the full CLI path through

    Caddy => auth validator => Sztabina

    without pre-cached credentials, against both INTERNAL and PRIVATE projects.

  • rk@tigase.net commented 3 weeks ago
    rksuma@Ramakrishnans-MacBook-Pro sztab % git checkout wolnosc
    git merge bugfix/SZ-140-git-push-command-fails
    git push
    git branch -d bugfix/SZ-140-git-push-command-fails
    git push origin --delete bugfix/SZ-140-git-push-command-fails
    Switched to branch 'wolnosc'
    Your branch is up to date with 'origin/wolnosc'.
    Updating f387833..35b73a0
    Fast-forward
     deploy/helm/sztab/caddy/Caddyfile     | 10 +++++++---
     sztabina/handlers/git_http_handler.go |  3 +++
     2 files changed, 10 insertions(+), 3 deletions(-)
    Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
    To https://tigase.dev/sztab.git
       f387833..35b73a0  wolnosc -> wolnosc
    Deleted branch bugfix/SZ-140-git-push-command-fails (was 35b73a0).
    remote:  
    remote: Create a pull request for 'bugfix/SZ-140-git-push-command-fails' by visiting:
    remote:     https://tigase.dev/sztab/~pulls/new?target=1325:wolnosc&source=1325:bugfix/SZ-140-git-push-command-fails
    remote:  
    To https://tigase.dev/sztab.git
     - [deleted]         bugfix/SZ-140-git-push-command-fails
    rksuma@Ramakrishnans-MacBook-Pro sztab % 
    
    
  • rk@tigase.net changed state to 'Closed' 3 weeks ago
    Previous Value Current Value
    In Progress
    Closed
issue 1 of 1
Type
Bug
Priority
Major
Assignee
Version
1.10.0
Sprints
n/a
Customer
n/a
Issue Votes (0)
Watchers (2)
Reference
SZ-140
Please wait...
Page is in error, reload to recover