Upgrade Git

If you’re still running a Git client that’s version 1.9.5 or older(!) you should really upgrade Git first (we’re up to 2.8+ now). There are a bug fixes (like this one) in newer versions of Git that should obviate the need to set http.postBuffer.

We had enough support requests from internal users and external customers hitting bugs in older versions of Git that we decided to add a server-side reminder to TFS/VSTS a few sprints ago:
M:\tmp\foo>git fetch
remote: Microsoft (R) Visual Studio (R) Team Services
remote: We noticed you're using an older version of Git. For a better experience, upgrade to the latest version at https://git-scm.com/downloads
remote: Found 4 objects to send. (6 ms)
Unpacking objects: 100% (4/4), done.

Check if you’re using a proxy or load balancer

If you’re using a terrible proxy that’s buggy or doesn’t support chunked encoding, you’ll see errors for larger pushes. The same thing can happen if you put on-prem TFS behind a misconfigured load balancer. If the same push succeeds when bypassing the proxy, or bypassing the load balancer (e.g. by pushing to localhost from the server itself), then fix your proxy or load balancer instead!

What if my proxy or load balancer is broken, but I don’t have any control over it?

This is the only scenario that I’ve seen where setting http.PostBuffer is useful for newer versions of Git.

Is setting http.postBuffer harmful?

AFAICT, it’s more unnecessary than harmful, but there are a few negative side effects:

Increasing it above the default may increase latency for larger pushes (since the client will buffer the HTTP request into larger chunks).

If you set it larger than the HTTP chunk size limit for your HTTP server (e.g. maxAllowedContentLength and maxRequestLength in web.config for TFS servers), then all pushes larger than the chunk size limit will start failing.

]]>https://blogs.msdn.microsoft.com/congyiw/2016/06/02/git-stop-changing-http-postbuffer/feed/0Why does cloning from VSTS return old unreferenced objects?https://blogs.msdn.microsoft.com/congyiw/2015/12/14/why-does-cloning-from-vsts-return-old-unreferenced-objects/
https://blogs.msdn.microsoft.com/congyiw/2015/12/14/why-does-cloning-from-vsts-return-old-unreferenced-objects/#commentsMon, 14 Dec 2015 17:44:00 +0000https://blogs.msdn.microsoft.com/congyiw/2015/12/14/why-does-cloning-from-vsts-return-old-unreferenced-objects/Note: “core Git” refers to the official base Git implementation, as opposed to Visual Studio or GitHub, or VSTS, which may involve non-standard implementations or behavior.

After running git gc locally, our local repo is now 5 MB, but git clone from visualstudio.com still returns 100MB. The old unreferenced blobs are still being sent down by the server.

How do we git gc (or some equivalent) on the server as well?

There are two issues here:

There is no equivalent to git gc on VSTS yet.

Our server preserves the history of every ref/branch update to Git repos, including deleted branches. This is analogous to the “reflog” in core Git. On VSTS, we expose the reflog via the REST API and the Branch Updates (i.e. pushes) tab in Web Access. Similarly to core Git, objects in the reflog are still considered to be referenced and will not be deleted by git gc. Core Git can eventually prune old reflog entries via git prune or git gc, but VSTS does not have that functionality yet.

Large fetches are expensive for the server to calculate, so we cheat a little.

Large fetches (and clones) have historically been very expensive in both core Git and VSTS due to the “counting objects” phase. http://githubengineering.com/counting-objects/ has a nice explanation of the problem, as well as how core Git and GitHub have (cleverly) improved the perf w/ bitmap indexes.

Unfortunately, VSTS does not have that perf fix yet. Instead, it cheats a bit and blindly streams back every object that exists on the server if the client has nothing and is asks for all branches and tags (e.g. for git clone). This is generally reasonable, until a user decides to dereference most of the objects in their repo to save space!

I suspect that the customer would not have minded the lack of gc in his scenario if we only sent reachable objects during clone.

Until these issues are fixed for VSTS, what workarounds are there?

Delete the repo from the server (EDIT: or rename it) and re-push it.

This works, but is sub-optimal. In the new repo, you won’t be able to see old pull request details, branch update history, and any links from other areas like builds or work items.

Run gitfiddler.cmd in a new command prompt. This script configures git.exe to use Fiddler as a proxy, and then waits for a key press. Once a key is pressed, the script clears the settings that it sets, and then exists.

Or the following error when checking in:

You can follow these steps to figure out which permission entries are blocking your operation. The instructions are for when you do not have Read permissions, but apply to Checkin permissions as well (just replace Read with Checkin)

These are the scenarios that can result in DOMAIN\Bob not having Read permission on $/p0/folder/a.txt

The Read permission is explicitly denied for $/p0/folder/a.txt for DOMAIN\Bob, or a group that he belongs to.

Note that group membership includes both direct and indirect membership (e.g. Bob belongs to subgroup which belongs to supergroup).

The Read permission is unset for a.txt and all parent folders that it’s inheriting permissions from, for Bob or any groups he belongs to. Unset permissions default to implicitly denied.

Note that you can hit this case when permission inheritance is turned off for a.txt or one of it’s parents.

The Read permission is explicitly denied on a parent of $/p0/folder/a.txt for DOMAIN\bob or a group that he belongs to. If this permission is not overridden at a lower level, $/p0/folder/a.txt will have inherit the deny entry.

If there are multiple applicable entries, denies will override allows. If Contributors are denied Read access, and Bob belongs to Contributors, giving Bob Read access will not actually give him Read access

The way to fix this is to either scope the deny down to a smaller group that Bob does not belong to, or turn off inheritance and leave permissions unset instead of using deny entries at all.

The following steps are only guaranteed to work if you authenticate as a member of the Team Foundation Administrators group (for the entire instance) or the Team Project Collection Administrators group. Being an ordinary user or a member of the Team Project Administrators group is insufficient. That is because if you do not have Read permissions, TFS will not tell you don’t have Read permissions. The exception is that the special Administrators groups are always allowed to see all paths and permissions. This is to keep Administrators from accidentally locking themselves out. Administrators will still get Access Denied errors, but they can at least see and fix the offending permissions entries.

In a Visual Studio Command Prompt, run tfssecurity /imx DOMAIN\Bob /server:http://SERVER:8080/tfs to get the list of groups Bob belongs to, directly, or indirectly.

In the same prompt, run tf permission $/p0/folder /recursive > out.txt, and open out.txt in a text editor that supports regex search (like Visual Studio):

Note the use of the /recursive flag

We are redirecting output to a text file because there may be ton of entries

Search for any paths that include the following entries:

“Inherit: No” where there is no explicit Allow: Read entry for the user or one of the groups he belongs to directly/indirectly

“Deny.*Read” (use a regex search) for Bob or any groups that tfssecurity showed.

[1] We block server side branching (which checks in immediately without pending changes in your workspace) if you don’t have recursive Read permissions on the source. Operations that result in server-side branches include:

tf branch /checkin

the VersionControlServer.CreateBranch() client object model call

Branching from a Branch Object in Source Control Explorer

We do this because:

If the user doesn’t have Read permission on an item in the source, he shouldn’t be allowed to branch that item

If TFS offered to skip those items, it could only tell the user that one or more items will be skipped, not which items will be skipped. The user doesn’t have Read permissions, so disclosing the affected paths could leak sensitive information.

If we allow the user to skip any items, there is no way for the user to figure out if anything important is missing from the new branch without talking to an Admin. In that case, the user might as well have the Admin perform the branch anyway.

If you pend a branch instead (a non-server side branch), then we silently skip files with no Read perms. I consider that a bug (and have filed it): it should show a warning that one or more items couldn’t be branched. Otherwise there is no way to trust that the branch is complete. Even an admin may be affected by explicit denies or disabled inheritance.

[2]I’ve heard on the MSDN forums that the TFS Permissions Sidekick from Attrice has a better UI for figuring out where you don’t have Read access, but I’ve never used it.