VSTS/TFS Agents behind a proxy

It often happens that I do write about my love affair with a web proxy systems. In the past years I shared many of these challenges through my blog. Following is another one that I came lately across and that is precisely, allowing the build agent tasks to gain the access to internet via a proxy.
Till now I have never noticed any necessity for setting explicitly the access via a proxy server on any of our build agents. We were using Azure App Service Deployment and other build/release tasks that did accessed an external web address and everything always worked without the need for any particular intervention. However, after the TFS 2017 Update 1, we noticed that our Azure App Service Deployment task started failing with the following error:

Error: connect ETIMEDOUT 104.45.0.17:443

Clearly the call towards the Azure services was failing. Couldn’t be anything else then our lovely proxy. Strange thing was that it was all working till before the upgrade. At this point I was curious why this was happening. Here is what I found.

With the TFS 2017 Update 1, a new version of the Azure App Service Deployment build/release task has shipped. This new task implementation is now based on Node.js handler and no Node.js applications were able to access the web. So obviously I went to setup the necessary to enable Node.js applications to get the access to the web. This was however not the right solution, also because build agent doesn’t use the Node.js engine installed on that machine, but executes everything via a version of Node.js that is shipped together with the agent itself. In order to get a bit more insight in this, I had to peak in the couple of MS Github repositories and luckily I found a reference in which I see that the Node.js HttpClient wrapper sets the proxy for all calls based on the environment variables called HTTP_PROXY and HTTPS_PROXY. Now, I tried adding those two variables and voila my build task started working again (after the build agent service restart).

So, in order to have your Node.js based tasks that do need to access Internet from a build agent behind a proxy, add two new environment variables called HTTP_PROXY and HTTPS_PROXY and set the value of those variables pointing to the proxy itself, like http://my.proxy.local:8080.

Do not forget to restart your agent and give it a go.
I hope this solves it.

Happy releasing!

TFS Tips from the Trenches

In the recent past I came across several interesting techniques to solve a non every day challenges with TFS. Not only that I would like to share those with you, also I would like to leave a trace about this as a note to my future self. I will list several short tips without a particular order.

Let’s start.

Proxy squared

I already wrote about allowing TFS to get the internet access via a proxy server in one of the past articles, TFS 2015 behind a proxy. It is not a very common situation to have a TFS server behind a proxy still in many enterprises it may be the case. My previous post shows how to let the TFS web application to access the web through proxy for TFS 2015 and it is also valid for the TFS 2017. However I missed to mention that there is another component on the application tier that also needs to be set and that is TFS Job Agent. You may ask yourself, why would TFS Job Agent have a need to access the internet? Well if you are trying to set up a web hook in your service hooks and your system needs to communicate with a machine that is out of your network, then the TFS Job Agent needs to be able to do so as it is him that actually sends the request generated by the chosen event. Luckily things are quite simple, move to the C:\Program Files\Microsoft Team Foundation Server 15.0\Application Tier\TFSJobAgent folder and open the TfsJobAgent.exe.config file. The following section needs to be added pointing to your proxy server

Once you saved these changes, you need to restart the TFS Job Agent. That can be easily done by executing the following from the command prompt with full administrator permissions:

net stop tfsjobagent

followed by

net start tfsjobagent

Now your web hook requests to an external party should succeed.

Browsing TFS from the AppTier machine fails in: ‘Unauthorized: Logon Failed’ error

In case you are not using the machine name to access your TFS server (by using an CNAME in DNS or accessing it via an A-Record that may lets say point to the NLB Virtual IP) you may discover a strange behavior of you Application tier server once it tries to access the service on the localhost. It is not an issue strictly related to the TFS and it has to do with the loopback check security feature that is designed to help prevent reflection attacks on your computer. You can read more about it at KB896861. This also can present an issue in case of TFS 2015 or earlier once you try to setup the Notification URL. A solution to this issue is quite simple and adding a value in the registry will solve it. The following PowerShell command will do the trick:

You need to set the Value to the URL you have chosen for your DNS entries. A server may require a restart in order to make the changes effective.

TFS DB’s under a SQL AlwaysOn replica

Obviously this is not a guide on how to setup an AlwaysOn replica on SQL Server and move your databases under the replication. That is a topic for a much longer post. What I would like to show you here, is what is necessary purely on TFS side in order to get your Application Tier to connect via the SQL Availability Group Listener to the cluster.

Following commands will make that happen:

TFSConfig RegisterDB /SQLInstance:TFS_LISTENER,10010 /databaseName:Tfs_Configuration

and

TFSConfig RemapDBs /DatabaseName:TFS_LISTENER,10010;Tfs_Configuration /SQLInstances:TFS_LISTENER,10010 /AnalysisInstance:TFSAS /AnalysisDatabaseName:Tfs_Analysis

TFSConfig command must be run from an elevated Command Prompt, even if the running user has administrative credentials. To open an elevated Command Prompt, click Start, right-click Command Prompt, and then click Run as Administrator.
TFSConfig tool is installed in the Tools directory – by default, this will be

  • TFS 2017: %programfiles%\Microsoft Team Foundation Server 15.0\Tools
  • TFS 2015: %programfiles%\Microsoft Team Foundation Server 14.0\Tools
  • TFS 2013: %programfiles%\Microsoft Team Foundation Server 12.0\Tools
  • TFS 2012: %programfiles%\Microsoft Team Foundation Server 11.0\Tools
  • TFS 2010: %programfiles%\Microsoft Team Foundation Server 2010\Tools

With the first command we will update the name of the server that hosts the TFS configuration database. SQL Instance parameter is pointing to the Availability Group Listener and not the actual instance of the server. 10010 is just the port on which the listener will replay (a non standard port in my case). In case you are on a version of TFS prior to 2017 you will need to include the /usesqlalwayson parameter, which on the TFS2017 is not anymore necessary.
For the second command we will redirect team project collection databases to be accessed via the SQL Availability Group Listener. Again if you are not running the TFS 2017 you will need to specify the /usesqlalwayson parameter at the end.
Make sure that you also specify correctly your analysis server as it is not accessible thought the SQL Availability Group Listener.
Once done do some failover tests and verify the correct functioning of your TFS instance.

More information about the TFSConfig tool can be found at Manage TFS server configuration with TFSConfig page.

Restore permissions for project administrators on Service Hooks

In case you upgraded your TFS instance from TFS 2013 or any previous version to TFS 2015/2017 it may happen in certain cases that the Project Administrators, a role which should have rights in creating and editing Service Hooks, will not be in place. You can add these permissions manually or you can use a tool provided by Microsoft to check the current situation and correct it is necessary. I made some changes to this tool giving you the opportunity to do that for all of the projects and collections on your instance. A fork of the tool can be found on GitHub at https://github.com/mmajcica/vsts-integration-samples.
Once you have downloaded and compiled the code, just run the command line tool with /Server parameter and specify the full path towards your service, like http://mytfs:8080/tfs. This will be sufficient for it to check and, if necessary, correct the missing rights. Same can be done for only a specific collection by using /collection parameter and passing in the path towards the desired collection, like http://mytfs:8080/tfs/DefaultCollection

Controlling and debugging TFS Jobs from DB

Often when you are checking your jobs and you realize that something went wrong, you need to analyze your issues in detail and retry failing jobs. This is usually done via a web services, however, it can also be done directly by querying the DB. I find the second approach often quicker and easier. Let me show you a couple of tricks and where is the necessary data located.

First thing first. Before we are able to do anything further we need to find the Id of the Job that we are looking to operate with. Let’s assume that I’m looking for ‘Reporting Service Path Rename’ Job, which in my case is failing.

Jobs can be defined in the Collection database as they can be defined in the Tfs_Configuration database. This specific one is on the collection level, so I will execute the following query on the collection DB.

At this point you should get back the JobId which we will use later for obtaining the execution history and to put a new job in the queue.
In my case the above query returned the following GUID: 6322B69A-04BD-47DF-9390-C3185ED59287

Now, on the Tfs_Configuration database you can now check the state of the above job with the following query:

This will bring us all of the failed runs for the given job in the chronological order. You can get valuable information from the result of this query. In particular I need to get the information about the JobSource which indicates the collection for which this job is failing
In order to get the collection ID <=> Collection Name mappings, you can check the follwoing table:

Let’s get to the point, lets trigger again the job from my example that was failing. This is the query that will create a new run for the given job:

As you can see, we used the job id that we retrieved earlier and the collection for which this is going to be triggered (as it is a collection specific job). Last thing left for us to do is to verify the state of the run and we can do that by checking on the tbl_JobQueue for all of the running jobs:

Now that you know the tables and SP’s in play, you can try it and proceed on your own.

Be very careful with modifying TFS DB! It is for sure a non recommended practice! 🙂

Conclusion

These are only some of many issues solved in the past for which I haven’t found a solution by simply asking Google. I hope these information will let you avoid spending hours in finding a valid solution to your TFS challenges.

Working with TFS in IntelliJ IDEA via VSTS Plugin

Recently Microsoft released a plugin for JetBrains IntelliJ IDEA in order to integrate this IDE with VSTS and TFS. Although Microsoft did a good job describing on how this plugin should be installed and described some basic settings, there are many non covered questions by the documentation provided. This is especially true when it comes to TFS and HTTPS. I will show you what is necessary in order to setup a connection for both Git and TFVC repositories on TFS.

Installing the plugin

In order to install the plugin, in the main screen of IDEA, choose Plugins from Configure menu:

2016-12-23-08_29_00

Once the plugin window opens, choose Browse repositories

2016-12-23-08_31_45-plugins

In browse repositories search for Visual Studio

2016-12-23-08_32_32-browse-repositories

If the Visual Studio Team Services plugin is not found, your connection may not be setup correctly. In case you are, as I am, behind a proxy, you need to click on HTTP Proxy settings button in the same screen (bottom left) and you will be presented with the following dialog

2016-12-23-08_30_10-http-proxy

Here you need to setup the necessary parameters (Auto detect proxy settings worked for me) and test your connection by clicking on Check connection button. Once done you will be prompted to enter an address that is behind the proxy so that internet connection can be verified. I used http://www.google.com/ for my test.

2016-12-23-08_30_38-check-proxy-settings

After confirming, you should see the following message

2016-12-23-08_30_48-check-proxy-settings

Now, back to Browse repositories screen, the Visual Studio Team Services plugin now should be found. Click on install and after the procedure finishes you will be asked to restart IntelliJ IDEA. Do so, restart the IDE.

2016-12-23-08_33_59-platform-and-plugin-updates

The plugin is now installed. In case this is not sufficiently clear, you can also follow the Microsoft guide Visual Studio Team Services Plugin for IntelliJ IDEA and Android Studio.

Be also sure that you have at least version v1.111.0 installed as I encountered issues with TFVC and previous versions of the plugin.

Importing the certificate

In case you are not accessing TFS through SSL connection you can skip this part. For Visual Studio Team Services plugin to connect to the TFS via https, that is using a self-signed certificate, no matter if you do intend using Git or TFVC, you need to import the certificate in the IntelliJ IDEA certificate store. To do so, export your certificate in the Base-64 encoded X.509 format. You can read about this in one of mine previous posts, Installing self-signed certificates into Git cert store.
Once you exported your certificate and let’s say saved it in C:\temp folder under the name tfs.cer, you need to open the command prompt and position yourself under the folder containing keytool.exe application. You can find it in the IntelliJ IDEA install folder, which in my case is C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.3.1\jre\jre\bin.

Once there you will then execute the following command

keytool -keystore "C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.3.1\jre\jre\lib\security\cacerts" -importcert -alias TFS -file "C:\temp\tfs.cer"

You will be then prompted for the keystore password. If you haven’t changed it, the default password is

changeit

After typing the password you will see the details about the certificate you are trying to import, and again you will be prompted to confirm that you trust this certificate. Type yes and you are done. Your screen should look like this

2016-12-27-11_16_12-administrator_-c__windows_system32_cmd-exe

Visual Studio Team Services plugin and Git

Before we even start, we need to be sure that a Git client is installed on our machine. From the Welcome screen choose Configure then Settings.

2016-12-29-15_08_58

In the settings window move to Version control -> Git pane and test the path to your Git client

2016-12-27-10_17_37-default-settings

In case all is good you should get back the version of your Git client

2016-12-29-15_09_41-git-executed-successfully

If this is not the case, or the path is wrong (change it and try again) or the Git client is not present on your machine. You can install Git for windows and you will find the necessary here.

In case you are using the https connection with TFS and it is based on a self-signed certificate do not forget to add that certificate into the Git cert store. This is something different then adding it to the Java cert store that I described in the previous chapter. You will need to do both of these steps. As I already described this procedure in detail, you can follow my other blog post Installing self-signed certificates into Git cert store.

Once the Git client is installed and certificate is imported, we can continue setting up the Visual Studio Team Services plugin. Get back to the welcome scree of IntelliJ IDEA and in the version control drop down choose Team Services Git

2016-12-23-08_34_54-welcome-to-intellij-idea

At this point a new dialog will be presented to you.

2016-12-27-11_05_09-checkout-from-team-services

Move to the Team Foundation Server tab and specify the address of your TFS server, then click connect. You will now be prompted for the credentials and if everything is ok, you will be show the list of available repositories

2016-12-27-11_18_08-checkout-from-team-services

You are now able to clone the repository of your choice and start working with it directly from IntelliJ IDEA.

Visual Studio Team Services plugin and TFVC

Before starting with TFVC, as for Git, we need an external tool. The tool in question is TF command line tool. It ships with the Microsoft Team Explorer Everywhere 2015 and you can download it here.
The file we are interested in is TEE-CLC-14.0.3.zip. Download it and unzip it in a folder of your choice. You should end up with something similar to this.

2016-12-27-12_15_39-tee-clc-14-0-3

Now, open the command prompt, move into the folder where you have extracted the TF command line tool and run the following:

tf eula /accept

If command succeeded and you haven’t received any error, you are good to go.

2017-01-02-09_01_28-administrator_-c__windows_system32_cmd-exe

Now back to IDEA. Open the settings panel.

2016-12-29-15_08_58

and move to Version Control -> TFVC pane. In the select path to executable field, enter the exact path to the tf.cmd command file located in TF command line tool folder.

2016-12-29-16_09_38-default-settings

Once done, press the test button and you should see the following message

2016-12-27-12_29_32-tfvc-command-line

Confirm all of the open windows and get back to the IDEA welcome page. Now you are ready to choose Team Services TFVC (Preview) version control.

2017-01-02-08_58_59-welcome-to-intellij-idea

At this point, same as for Git, you will be prompted about the connection towards your TFS. The following dialog will be shown.

2016-12-27-11_05_09-checkout-from-team-services

Move to the Team Foundation Server tab and specify the address of your TFS server, then click connect. You will now be prompted for the credentials and if everything is ok, you will be shown the list of available TFVC repositories.

2016-12-27-13_18_30-checkout-from-team-services

You can now create a new workspace directly from IDEA and start working with your TFVC repositories.

Troubleshooting

There are a couple of common issues you may encounter in following what I just described. I will tell you about the most common ones and how to overcome those.

In case you see the message in the following screenshot

2016-12-27-11_05_28-checkout-from-team-services

You are probably facing some issues with the certificate. Make sure that you exported/imported the certificate correctly.

During the test of the TF command line utility, you may encounter the following exception

2016-12-27-12_28_41-tfvc-command-line

In may case it turned out to be a problem with allocating the heap memory from TF process. I could clearly see in the log file the following:

2016-12-22 13:14:08,949 [ 17642] INFO - ugin.external.commands.Command - 167680800(ns) - elapsed time for add -noprompt -?
2016-12-22 13:15:46,592 [ 115285] INFO - lugin.external.ToolRunnerCache - getRunningToolRunner: toolLocation={0}
2016-12-22 13:15:46,592 [ 115285] INFO - lugin.external.ToolRunnerCache - getRunningToolRunner: slow version - null
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - ToolRunner.start: toolLocation = C:\Utils\TEE-CLC-14.0.3\tf.cmd
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - ToolRunner.start: workingDirectory = null
2016-12-22 13:15:46,592 [ 115285] INFO - alm.plugin.external.ToolRunner - arguments: add -noprompt -?
2016-12-22 13:15:46,716 [ 115409] INFO - ugin.external.commands.Command - CMD: Error occurred during initialization of VM
2016-12-22 13:15:46,717 [ 115410] INFO - ugin.external.commands.Command - CMD: Could not reserve enough space for 2097152KB object heap
2016-12-22 13:15:46,731 [ 115424] WARN - ugin.external.commands.Command - CMD: parsing output failed
com.microsoft.alm.plugin.external.exceptions.ToolBadExitCodeException: KEY_TF_BAD_EXIT_CODE
at com.microsoft.alm.plugin.external.tools.TfTool.throwBadExitCode(TfTool.java:109)
at com.microsoft.alm.plugin.external.commands.Command$1.completed(Command.java:155)
at com.microsoft.alm.plugin.external.ToolRunner$ListenerProxy.completed(ToolRunner.java:289)
at com.microsoft.alm.plugin.external.ToolRunner$ProcessWaiter.run(ToolRunner.java:327)

A workaround for this issue is to modify the tf.cmd file by specifying a lower -Xmx parameter. By default it is set to 2014MB however a 1024MB also worked well for me.

Another issue with this version of Visual Studio Team Services plugin is in changing in between the Team Services TFVC (Preview) and Team Services Git, or viceversa. You can read more about it here, TfsAuthenticator hangs the IDE.

No matter the issue, you can always find more information about it in the log file. The log file is located in my case in C:\Users\majcicam\.IdeaIC2016.3\system\log. Adapt this path to your case. The log file name is idea.log.
Visual Studio Team Services Plugin settings can be found in vsts_settings.xml file under C:\Users\majcicam\.IdeaIC2016.3\config\options folder.

Useful links

On visualstudio.com you can read more about this topic and see some useful how-to video.

  1. Installing Visual Studio Team Services Plugin for IntelliJ IDEA and Android Studio
  2. Using Visual Studio Team Services Plugin for IntelliJ

With all of these information I do hope you can get on going with the plugin and boost your productivity.

Happy coding.