Menu

Xcode Bots, hosted git repositories, and automated TestFlight builds!

Added steps to import the app’s Distribution Identity into the Xcode Server keychain

Fixed problem with Post-Action archive script that would post n-1 versions of the build to TestFlight

This kind of post represents why I started this blog: I spent days banging my head against the wall trying to get this all working, and no one else should have to endure the same.

This post outlines the major steps and gotchas surrounding how to:

Create a Bot that fetches source from a 3rd party Git repository

Builds, tests, and archives a project successfully

Automagically post an fresh build to TestFlight for automatic distribution every day to testers (this is as cool as it sounds)

Here are some prerequisites to consider:

You have OS X 10.9 with OS X Server installed.

I used a dedicated Mac Mini; so you may need to translate accordingly if you are using your local machine (though I don’t really recommend it).

You are familiar enough with provisioning profiles to be dangerous.

You have a working 3rd part git repository set up for your workspace. I’m using Assembla, but it should all be about the same with github, etc.

You have a TestFlight account and you know how to use it.

I will be using our Unwired DocLink product, a universal iOS content distribution app, as the model for this post.

As a disclaimer: this is all working for me… I did a lot of head banging and tweaking things to get stuff to work, most if not all of which is described here. Some stuff should have just “worked”, but didn’t at the time of this writing. I would like to keep this article updated if I missed something, so please post your comments below so I can update this article as needed.

And here we go!

Setting up Xcode

First up, you need to do some config with Xcode.

Create a new Build Configuration

Before you create a new scheme, you may want to set up a new Build Configuration. Projects created with Xcode come ready out of the box with Debug and Release builds. I create a duplicate of Release and named it TestFlight. You do this in the Project info screen as shown here:

Create a new build configuration named TestFlight

Next you are going to want to configure your distribution provisioning profile for the build you just made. It is helpful to filter the list down using the search bar at the top. I am using an enterprise distribution certificate, but you can specify an AdHoc one here as well.

Setting the Provisioning Profile

Keep in mind you will need to make sure all of your other build settings are correct as well, including making sure the correct code signing identity is specified in the build settings as well.

Create a new Scheme

You need to create a scheme that your Bot will use. You could technically use your default one, but to keep things a little cleaner, I created a new one. Simply create a duplicate scheme of your working one using the gear in the lower-right of the Manage Schemes window.

Create a Duplicate Scheme

Name the scheme something like “<ProjectName>_CI”. I would avoid white space (unlike my example) because things get a little hairy with paths later and that is one less thing to worry about.

With the newly created Scheme, make sure to check the “Shared” checkbox.

Highlight your newly created scheme and click Edit…

In this window, everything is likely fine by default with the exception of the Archive build configuration. There are a few things to do there, which needs its own section!

Configuring the Archive Scheme Build Process

There are two things you need to do here: configure the proper Build Configuration and specify your TestFlight deployment script.

First, make sure the Archive build configuration is set to TestFlight; the configuration you had just created.

Archive Build Configuration

And now here is where things get fun! Click on “Post-actions”, and if nothing is in there, click the ‘+’ at the bottom to create a “New Run Shell Script” action.

Below is the script you should use and substitute your values as needed. An explanation of important aspects of it are explained afterwards.

Whew! There are a lot of little nuggets in there worth explaining:

UPDATE: My original script used a “/Latest” symbolic link that the Bot would create, however this is created AFTER the integration has completed. Therefore, your builds that were uploaded to TestFlight were version-1. By using the ‘ls’ command, you effectively get the working script. Tangentially, the Archive.xcarchive.zip file does NOT exist at Post-Action time, but the folder does exist. The above script reflects both of these updates.

Your signing identity is the name of the certificate / key in your Keychain. It should be an EXACT string match, as shown in my Keychain.

Distribution Identity Name

You’ll notice some hard-coded paths. These are some of the paths used by the Xcode build server that don’t exist on your normal workstation. Some of these paths have been painfully crafted to make work after lots of head-banging.

The /tmp folder is used as a working directory to do intermediary work. It is cleaned up every time a build is performed.

This is all working as of the very first GA release of the Xcode Server; later releases may change paths that would require modification to the above script.

Make sure your build is sane

With the newly created scheme selected (next to the run and stop button in Xcode), select iOS Device (or any connected device) and go to the Product menu and select Archive.

Make sure the archive process works, because if it doesn’t work on your development station, it sure as hell isn’t going to work on the Xcode server :). The TestFlight upload process will fail because the specified directories don’t exist on your local machine (unless the Xcode server is on the same machine). However, Xcode should open organizer and a new TestFlight release build should be available as you would expect.

If so, commit your project changes to your git repository. The Xcode server will need all of those settings we just changed in the project file to work correctly.

If you get build errors, Google is your friend. There are simply too many build configuration problems to try explain here. Once you get it working, however, read on!

Prep the Xcode Server

If you haven’t already, make sure you have the Xcode Server running on your Mac OS X server. That set up is pretty straightforward.

There is no need to create any repositories here! That will be handled when you create the bot on your workstation.

However, do make sure you allow HTTPS access to repositories:

Repo Settings

Installing Distribution Provisioning Profiles

In theory, at this point you should be done. However, at the time of this writing, essential distribution provisioning profiles are not installed when you configure teams in Xcode server. This means your Xcode server has no way of properly signing your app, resulting in build errors time and time again during the archive phase.

This means you have to manually download the .mobileprovision file(s) and move it/them to the appropriate location on the server. The good news is this is pretty easy.

You have to run the copy using sudo, because of tight permissions on the folder. That said, there is no need to change permissions.

Installing Your Distribution Identity into the Keychain

Another thing the Xcode server doesn’t (can’t?) do is post your app’s iPhone Distribution identity (certificate and key) into the keychain of the Xcode server itself. Remember that this Distribution Certificate (aka Signing Identity) is tied to your Distribution ProvisioningProfile. Without this, your app will not compile and you’ll get an error like this:

With Xcode 5, the Distribution Identity is created through Xcode automatically when you are the agent for the account and a Distribution Certificate doesn’t yet exist. If one already exists, and it isn’t in your keychain already (see below), you may have to either a) manually download the certificate from the iOS Developers portal or b) hunt down the person who has the identity and have them send it to you.

All that said, here are the steps to extract the Distribution Identity from your development machine’s keychain (assuming it was created there), and transported to your Xcode server.

First up, open the Keychain app on your development computer. Select the login keychain and then click My Certificates under categories. In the Search field, type “iPhone Distribution” and you will see your distribution certificates float to the top:

Locating your Distribution Certs

Now a key (pun kinda intended) thing here: make sure you have the little arrow to the left of the name of the certificate. If you do NOT have it, then you DO NOT have the private key for the distribution certificate and things are not going to work (this means you have the certificate, but not the identity). You will need to find the person/computer that DOES have this private key and perform this task from there.

Now, with the arrow opened or closed (I opened it to show you the key, your may be named different) right-click the certificate row and select Export. From there, provide a strong password and save the resulting .p12 file somewhere on your hard drive.

Export the Identity

Note that sometimes the Export option doesn’t show up (at least in OS X prior to 10.9, it may have been fixed). Simply toggle the arrow, click on something else and back, etc. and it should then show up).

At this point, the .p12 should have been saved where you specified. Now transfer it to the Xcode server using your file transfer method of choice: SSH, AirDrop, FTP, GOPHER… whatever floats your boat.

On the Xcode server, locate the transferred .p12 file and drag-n-drop it onto of the Keychain Access app (/Applications/Utilities). You will be presented with a dialog to import the certificate. Select the system keychain then click Add. Provide the strong password you created when exporting the .p12 file, and if all goes well, you will see the identity in System -> My Certificates.

Import the Identity

Guess what, your Xcode Server now has what it needs to build archives for your app!

Creating the Bot

Alright, now this is where things are a little weirder than you would expect them to be.

In my case, I use SSH to access my git repository. However, I cannot for the life of me get Bots configured with SSH to properly pull from the repository. It may be an Assembla thing, but despite my best efforts, I cannot get it working. Instead, I use HTTPS (with username and password) to access the source.

Now here is a weird thing (that may be fixed in a later release). You can only create a Bot using the same git mechanism you have configured for your current workspace! This means that since I am using SSH to access my repo, the only auth form I can use when creating a bot is with SSH (there is no option to switch to HTTPS).

If you can get SSH authentication to work properly on your Xcode server with the specified repo, please share! Otherwise, I am going to assume you will do this HTTPS workaround like I did.

So, how do you use HTTPS? Checkout your project into another folder using HTTPS. You only need to use this for the bot setup, then you can delete this copy from your development machine because the server will now have all it needs to properly authenticate.

In my case, I closed out of my DocLink workspace and basically checked out the same workspace using HTTPS instead of SSH on a different folder on my HD. You can use the Check Out option within Xcode (and create the Repository entry via HTTPS there), or you can checkout via command line. Either way, when you are done you should be looking at an identical copy of your workspace, only using HTTPS with your source control.

Great, now lets create that bot!

The Bot Wizard

Go to the Product Menu and select Create Bot…

In there, you are going to want to make sure to select the new scheme you had made. If it is complaining about not having the shared scheme, make sure you had checked the “Shared” checkbox in the Manage Schemes window and committed your changes to the git repo.

Your Xcode server should appear in the Server list, or you may need to create the server entry first from the menu.

Check the Integrate Immediately so a test run can complete and we can see what is going on.

Initial Bot Config

Click Next.

You may be prompted for your git credentials next (this may be out of order; I’m not going to break my build for the sake of this post, sorry :). When you are, complete the form as appropriate:

Assembla Repo Config and Auth

Click Next.

You can schedule the bot to run whenever you’d like, but I would recommend once a day. The reason being is that you can quickly end up with TONS of TestFlight builds if you have the bot run on every commit, making it somewhat challenging. I run at 3:00am daily, so testers have a new version to work with in the morning.

Bot Schedule Config

Now for another painful lesson that will probably be fixed in a later release: there may be multiple iOS simulators for a give form factor, but not all of them work… This turns into a bit of a guess gaming (you can change these settings after bot creation through the Xcode Server’s website), but rest assured, your tests won’t run if you specify this wrong. So, in short, do not select all iOS Simulators or Devices if you have duplicate entries when selecting Specified Devices. Below is the screenshot of the multiple entries. It seems like the copies lower in the list are the ones that work, though they are shown differently in the Bots Settings via the Xcode server website, so I can’t say for sure.

Device Selection

On the next screen, set up notification options to your preference. Then click Create Bot.

If all goes well, you get a nice big green checkmark! Bot created!

Verifying the Integration

Now we’ll want to make sure things worked alright (if you are lucky, the will!)

Use the Xcode Server website. Usually it is just your machines root URL (e.g. http://ci.unwiredrev.com for our machine). You will see a screen like this (though maybe not with everything because you may not have run successfully before):

Xcode Bots Webpage

It should show “Getting Sources” or “Integrating”.

Clicking on a Bot’s name will reveal details. After some time of things working well, you should see something like this:

Bot Status Details Screen

You can see where things were not working so well…

When an integration is done, it should show completed for the most recent integration, ideally without any errors.

The “Logs” tab is really useful, as it give full logging details for each integration. Our TestFlight script actually writes out to this log during integration, so any errors with the upload show show there. Below is an example of a successful TestFlight upload:

Build Logs with TestFlight Upload Success

I’ve found Sometimes builds will fail for no reason. Rebooting the server has fixed this a few times for me… This was during the beta period, and may not be a problem now.

You also may want to disable Testing and Analyzing while you are figuring this out (in Bot Settings) to save time on the integrations just to make sure the archive is posted to TF correctly.

I think that’s about it! I’ll update this post as I receive feedback, so please comment if this process has worked for you or not. Good luck!

Thanks you so much for your great post. I’ve been having problems setting up my Xcode configurations for CI. Before reading your post, I was getting a “no such provisional profile found”. After copying them as you said, I don’t get that error anymore but I get couple of different errors:

I just moved to another computer, so maybe I’m not setting the developer certificates right? Everything runs well in my devices locally and every build is successful. I only ge this when integrating with the Bot.

Ah, totally forgot: you need to import the distribution identity to the keychain on the Xcode server.

I’ll update this post, but basically you need to find the identity on the computer that had “requested” the distribution identity (it should have the format of “iPhone Distribution: .” Make sure there is the little “arrow” to left of the certificate, that reveals the associated key. Right click the certificate an choose Export. Save it out and transfer it to your Xcode server and import into your System keychain.

More screenshots to come. The second code signing error is an effect of the first one. It should go away after importing the identity… Let me know how it goes!

So my project is using Pods and it is working fine… I have little to no experience with Pods so it will be tough for me to know what is normal or not. All I can say is that the Pods component worked right out of the box for me.

I have a PODS_ROOT setting in the User-Defined section of my apps main Build Settings (${SRCROOT}/Pods), which is coming from the Pods.xcconfig file that is part of my main app project bundle.

@Matt,
Do you check your Pods folder into source control? If you do, yeah you shouldn’t have any problems with it. But some people (including my team) prefer not to check the Pods folder and Podfile.lock into the source control but do a ‘pod install’ before every build, that’s where all kinds of issues come in.

Great post. Setting up the bot with ssh authentication was no problem for me. And it was not necessary to use the same authentication on the development machine. For testing the Xcode CI possibilities I have created a demo project with ssh authentication on the bot and https authentication on the development machine.
I will do a post about my steps, but it will be in german because of my bad english. Maybe my screenshots can help you.

However, I ran into a ton of issues with keychain access, mainly the famous “user interaction is not allowed”. After trying out different possible solutions, I found the only way to get it to work is creating temporary keychain and point codesign to use it. Here’s what I have to add to the post-actions script, in case some one else in the future come across this post and have the same issue:

As per http://stackoverflow.com/a/14761060/1252368 the issue can be solved without scripting if you have access to the server’s UI – which you most certainly have when you use the Keychain Access application to install the certificate and identity.

After you put the certificate and key into the system keychain, select the key and shows its properties via the context menu’s “Get Info” option. Then add /usr/bin/codesign to the list of allowed applications that can use the key without asking for permission first. You will have to enter your password one or two times to confirm, but once that is done, codesigning will work (just did it :))

Got this error too but Daniel Schneller’s post above helped me out. (I think this is what happened: the key permissions looked ok after import but where stripped at some stage, once I added /usr/bin/codesign back it was all good.)

This has been a major headf*k even with this great guide, can only imagine what it had been like for me without it. Thanks Matt!

Excellent article. You revealed an issue that has aged me nn-years. I could _not_ create Bots from within Xcode, only via web interface. But your hat dance to make a temp working directory and clone a repo via HTTP was the trick !!!

Life is better, finally after weeks/months of belligerent Xcode behavior refusing to create a bot (it insisted that my Workspace was not under source control when in fact it was).

I have another issue though, wondering if you have any insight.

First I have a workspace with 1 app, and three of my own static libraries. Each library has its own git repo (all git repos in house on a mac mini OS X server).

The shared workspace scheme explicitly lists each of the static library schemes as dependents. Each library scheme is also shared.

The Bot runs, but fails on the link pass naming the first static lib as not found.

I’ve made individual Bots for each of the static libraries. These individual Bots work fine, even run the unit tests. But of note is the 1 to 1 relationship Bot-to-Repository.

Also if I go to the server and manually pull the trees in Xcode, the workspace builds and runs fine.

I’m also quite frustrated with Bot configuration (its hopelessly dumbed down imho). There does _NOT_ seem to be a way to inform a Bot that there is more than _one_ repository to pull.

In my case I have 3 static libs in the workspace that I associate with the Bot. I’ve scanned the mammoth /Library/Server/Xcode/Data/BotRuns/Latest logs and am unable to ascertain if those dependent repos are being cloned and built.

Have you or anyone you know successfully used a Bot on a Workspace with > 1 git repository ?

Hi,
Just wondering if anyone has had success defining a Bot with a workspace that has more than 1 source repository ?

I use 3 static libraries in the workspace, the scheme is shared and all builds well from within xcode. Each library has its own git repository. All repositories, the App and all libraries are hosted on the same CI server running the Bots.

Anytime I run the Bot it only checks out the main App repo none of the library repos. Consequently the Bot always fails to compile.

Hi,
Just wondering if anyone has had success defining a Bot with a workspace that has more than 1 source repository ?

I use 3 static libraries in the workspace, the scheme is shared and all builds well from within xcode. Each library has its own git repository. All repositories, the App and all libraries are hosted on the same CI server running the Bots.

Anytime I run the Bot it only checks out the main App repo none of the library repos. Consequently the Bot always fails to compile.

Program /usr/bin/codesign returned 1 : [iPhone Distribution: COMPANY_NAME (XXXXXXX): no identity found
]

I am pretty confident that the identity is installed correctly. When I manually piece together the xcrun command that the script is executing, and run it from the terminal on the server, it seems to run successfully and the IPA is created in the tmp directory. Any ideas what could be causing this?

The first time I tried this provisioning and signing worked perfectly without me having to do anything. Unfortunately this was just some preliminary testing on a very old machine that was taking over an hour to do a single integration.

I’ve set up a new CI on a newer machine and am in the process of resolving the provisioning issue at the moment. Thanks for your insights.

I had no issue setting up SSH to the remote git repo. If you haven’t figured this out let me know and i’d be glad to help out.

The provision was copied to /Library/Server/Xcode/Data/ProvisioningProfiles/ via sudo, but _teamserver had no read access to it. Make sure that _teamserver has read access to the profiles by changing ownership with this (94:94 is the uid and group for _teamserver):

sudo chown -R 94:94

My builds are still reporting Archive Failed after every run, but Archives are being produced.

I also set out to automate the upload process (to a Nexus in my case). Close to the end, I came across your post which helped me to cross the finish line (IPA creation) – thanks a bunch!

Being in the post archive action, there are a few environment variables you can use, for example ARCHIVE_PATH or the CODE_SIGN_IDENTITY from the build settings (I dumped env into a file to see what’s available).

I also added user defined build settings (NEXUS_FILENAME_PATTERN for instance) to control the behavior in one place only (the build settings). This also gives me a neat way of defining a different behavior for different configurations.

I still get the following error and I don’t know how to solve it. Here is the error log:
/Library/Server/Xcode/Data/BotRuns/Cache/716e8e03-d441-4f4c-b7bf-3f3ab34306df/DerivedData/Build/Intermediates/ArchiveIntermediates/AbaTrack/InstallationBuildProductsLocation/Applications/AbaTrack.app: User interaction is not allowed.
Command /usr/bin/codesign failed with exit code 1

Does anybody had the same problem and does anybody knows how to solve it?

For all of you who did receive the same error:
User interaction is not allowed.
Command /usr/bin/codesign failed with exit code 1

This error happens when the following setting is not made manually. As described in one of the posts above you have to put the mobileprofile for code signing in the system keychain. The certificate is called iPhone Distribution:. This distribution certificate must have a private key! Open the Settings dialog either by double click on the private key or through the context menu>Get Info.
Then go to Access control and put /usr/bin/codesign in the table. This allows the codesign application to sign your built application. After that everything should work without the meaningless codesign error exit code 1.

Daniel, did you set the perms of 94 (_teamserver) as per the comments above? Did you use the command line version of the keychain import rather than the Keychain Access gui tool? I had to do both of those things for it to work for me.

Excellent article Matt, saved me a lot of time. I finally have everything running pretty well but a few needed tweaks remain.

One of the largest issues is that my Testflight upload will still proceed even if the test suite fails. Before I dig in to a way to configure around that, thought I’d check to see if anyone has a solution they would be willing to share.

Thank you a bunch pertaining to expressing that with all of individuals that you know what you will be communicating approximately! Added. Kindly likewise seek advice from my site Equates to). We will have a weblink alternative set up among us

I use sourcerepo.com as SVN server. My bot complains : error validating server certificate for “https://myproject.sourcerepo.com:443″ – the certificate is not issued by a trusted authority. Use the fingerprint etc….

How can I make the XCode Server accept this svn server and retrieve sources ?

Hi Frank,
In your bot’s source control log (viewable via web browser), you should see the following. The last sentence is the one you want.

SSH Known Hosts file path is located at /Library/Server/Xcode/Config/ssh_known_hosts
SSH strict host checking is enabled (you can disable this by editing the SSHStrictHostKeyChecking key in /Library/Server/Xcode/Config/xcsbuildd.plist
Untrusted HTTPS certificates is disabled (you can enable this by editing the TrustSelfSignedSSLCertificates key in /Library/Server/Xcode/Config/xcsbuildd.plist

There are books, home study courses, workshops and other means
of information, to gain the knowledge required to become the alpha leader.
It also has LED flash that illuminates when the need arises.
With the new Jelly Bean, you will be able to expand notifications
and get a preview on the recent e-mail message.

Howdy! I know this is kind of off topic but I was wondering if you
knew where I could find a captcha plugin for my comment form?
I’m using the same blog platform as yours and I’m having problems finding one?
Thanks a lot!

Great tutorial. Here are some things that I think are missing, if you want to add them (I just installed my first xcode bot svr on 10.9.2, on 26. March 2014):
– provisioning profiles: the directory is missing, you first have to sudo mkdir /Library/Server/Xcode/Data/ProvisioningProfiles/
– it would be handy if you could write the default location of xcode provisioning profiles, because it is very hard to find them. Luckily I found a copy of mine lying around!
– please include a general section that explains the differences between build schemes and build configurations, something like this: build configurations allow you to have separate code signing identities and provisioning profiles. Build schemes allow groupings of actions based upon build configurations, which bots can then use to perform the actions.
– I had to not only generate separate build configs/schemes like you did, but also parameterize the bundle identifier. Might be worth having a note on that: project -> each target -> info -> change bundle idenfitifier to ${CUSTOM_BUNDLE_ID} … then in each target -> build settings -> user-defined (at bottom), add CUSTOM_BUNDLE_ID and define the bundle identifier per build configuration as desired
– when you add new build configurations, you break any code that uses the preprocessor macros based on the name of the build configuration. To fix this: go project -> Build settings -> Apple LLVM 5.1 – Preprocessing section (or, search for “preprocessor macros not used in precompiled” -> set the duplicated build config to use the SAME value as the build config it duplicates
– as others have noticed, you must give codesign access to the code signing identity that you imported into the System keychain. What I don’t see noted here is that you can do this in the Keychain app UI itself: keychain -> System -> choose category “Keys” -> find your key (mine is an enterprise private key) -> Access Control -> add /usr/bin/codesign

… sorry I forgot one: it is probably worth mentioning that it is NOT WORTH TRYING TO GET SSH-KEYS & XCODE SERVER & GITHUB all working together. They just don’t like being in the same room — after spending more than 2 hours on this, including generating keys, converting formats, importing into various keychains and reading countless articles, I can say that it just doesn’t work (for me)
😉

Ahoy,
thanks Matt for this helpful website BUT …. I have one question:
You mentioned that you have to copy Provisioning Profile from local machine to Osx Server…

1°) How about local machine IS the OSX Server ? (I think it’s okay)
2°) How about wildcard PPs ? Even if i cp them,It seams my Server doesn’t see them for dev’ build. We are many devs/projects in the company so we have to use wildcards pp…

You advice ? Anyone who had experience about this problem ?

Thanks dude again for these tips, you helped a team about Remote Continuous Integration =)

1. Use a dedicated mac mini for the CI process. The server eats up a lot of resources and slows down the dedicated machine considerably (my dedicated machine is a late 2012 mac mini, quad core with 8gb ram).

2. Choose your bot test devices wisely. Never had an issue when using a real iOS device, while using simulators, I got errors or never-ending integration phases.

3. Don’t expect your archives to appear in the XCode organizer

4. The CI process is being performed in a protected storage area, and scripts which are creating files or folders, may fail because of write access issues.

5. I found that the “/tmp” directory (used in the testflight upload script) is only created after the archive process has finished. So creating scripts which use it, before/after the any other phase (build, test, profile…) might cause issues.

7. Make sure you commit all changes to the OSX Server, before integrating. That’s because the first thing the server does is to clone the repository, before the other phases like build/test/archive. The same goes for any pre/post action scripts to any XCode schemes.

The TestFlight upload script as of May 26 2014 does have minor inconvenience on my build server. It does not remove the previous IPA file and hence it uploads the same build again, which I see because my build number is not incremented and TestFlight adds a build number (#Number) . The change I needed was:

Great article, it helped me setting up something similar. Instead of Testflight I’m using a script to upload to our own API.

One thing that I think that could be improved: Xcode sets a lot of environment variables for you, including the Signing Identity and Provisioning profile that are set in the project settings. Using these would save you the trouble of setting these when calling the script, and that’s always a good thing (less typo’s). look for `CODE_SIGN_IDENTITY` and `PROVISIONING_PROFILE`.

(I printed out all environment variables that are available to find this out, and there are just under 700 in my case. There may be more useful ones I missed)

Google Voice is not, a carrier replacer(Well at least not yet).
Naturally, you cant combine a Google Adwords promotional code with another promotional code or any other offer from
Google. If there is already lots of Google Adsense put into that certain site, put yours on top of
aall of them.

Hi, I do believe this is an excellent blog. I stumbledupon it ;
) I am going to come back once again since i have saved as
a favorite it. Money and freedom is the best way to change, may you be rich aand continue to help other people.

There are all sorts of ports, buttons and assorted controls around the unit’s perimeter,
with the volume controls, a power button, a micro –
SD slot and a stylus on the right side. Ravensword: Shadowlands: A beautiful RPG with
a world to explore, various monsters to fight, and plenty of medieval
& magical equipment to fight them with. In past mobile phones are mainly used to make call and on very high cost but now it.

Hi do you know that you can increase your conversion ratio couple of times and earn additional
bucks every day. There is incredible landing
pages tool. It is very easy even for noobs, if you are interested simply search in gooogle:
pandatsor’s tools