Script to import repositories to Bitbucket

Question 1

I needed a script to import repositories from other Bitbucket accounts to my own Bitbucket account. I have some 250 repositories in my account and I wanted them to move under Team (which is same as Organization in Github). Though Bitbucket has APIs, I couldn't find any API to import, and instead they provide a HTML form on their site. When I couldn't get my pure Python script to automate and fill the form, I decided to go with Selenium IDE. The following Python script generates Selenium IDE script, which I later run on the IDE.

This is how this script works:

It takes a REST call to Bitbucket, to find all the repositories
The Bitbucket provides a paginated response, the response includes repo URL which is just enough
I make a series of calls to get all repository URLs
Once I have all the URLs, I fill in the Selenium script template and write it back to a file

This script also can be used to import repositories from Github/or any other version control server by importing the generate_sel_script function.

import json
import requests
from requests.auth import HTTPBasicAuth
from templates import sel_script_template
from templates import command_template
from settings import *
def generate_bb_repo_urls():
 def make_rest_call(url):
 auth = HTTPBasicAuth(SRC_AC_USERNAME, SRC_AC_PASSWORD)
 return requests.get(url=url, auth=auth).json()
 seed_url = 'https://bitbucket.org/api/2.0/repositories/' + SRC_AC_USERNAME
 response = make_rest_call(seed_url)
 bb_repo_urls = []
 while('next' in response.keys()):
 for repo in response['values']:
 bb_repo_urls.append(repo['links']['html']['href'])
 response = make_rest_call(response['next'])
 return bb_repo_urls
def generate_sel_script(src_repo_urls):
 commands = "".join(map(lambda url: command_template.substitute({
 'bb_username': SRC_AC_USERNAME, 
 'bb_password': SRC_AC_PASSWORD, 
 'repo_url': url,
 'owner_value': IMPORTER_ID
 }), src_repo_urls))
 with open('selenium-script.txt', 'w') as sel_file: 
 sel_file.write(sel_script_template.substitute({
 'bb_username': IMPORTER_BB_USERNAME, 
 'bb_password': IMPORTER_BB_PASSWORD, 
 'commands': commands
 }))
if __name__ == '__main__':
 generate_sel_script(generate_bb_repo_urls())

settings.py:

SRC_AC_USERNAME = 'bitbucket'
SRC_AC_PASSWORD = 'ohlongjohnson'
IMPORTER_BB_USERNAME = 'bitbucket'
IMPORTER_BB_PASSWORD = 'ohlongjohnson'
IMPORTER_ID = '007007'

templates.py:

from string import Template
sel_script_template = Template("""<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="https://bitbucket.org/account/signin/?next=/repo/import" />
<title>New Test</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">New Test</td></tr>
</thead><tbody>
<tr>
 <td>open</td>
 <td>/account/signin/?next=/repo/import</td>
 <td></td>
</tr>
<tr>
 <td>type</td>
 <td>id=id_username</td>
 <td>$bb_username</td>
</tr>
<tr>
 <td>type</td>
 <td>id=id_password</td>
 <td>$bb_password</td>
</tr>
<tr>
 <td>clickAndWait</td>
 <td>name=submit</td>
 <td></td>
</tr>
<tr>
 <td>pause</td>
 <td>100</td>
 <td></td>
</tr>
$commands
</tbody></table>
</body>
</html>
""")
command_template = Template("""<tr>
 <td>pause</td>
 <td>1000</td>
 <td></td>
</tr>
<tr>
 <td>open</td>
 <td>https://bitbucket.org/repo/import</td>
 <td></td>
</tr>
<tr>
 <td>pause</td>
 <td>100</td>
 <td></td>
</tr>
<tr>
 <td>open</td>
 <td>https://bitbucket.org/repo/import</td>
 <td></td>
</tr>
<tr>
 <td>type</td>
 <td>id=id_url</td>
 <td>$repo_url</td>
</tr>
<tr>
 <td>click</td>
 <td>id=id_auth</td>
 <td></td>
</tr>
<tr>
 <td>type</td>
 <td>id=id_username</td>
 <td>$bb_username</td>
</tr>
<tr>
 <td>type</td>
 <td>id=id_password</td>
 <td>$bb_password</td>
</tr>
<tr>
 <td>select</td>
 <td>id=id_owner</td>
 <td>value=$owner_value</td>
</tr>
<tr>
 <td>clickAndWait</td>
 <td>xpath=(//button[@type='submit'])[2]</td>
 <td></td>
</tr>""")

What I am looking for:

Any general code improvements/comments
How do I avoid from settings import *? (or is it okay for import * in this case) I have two options, import settings or five import statements for each variable.
Is it possible to avoid this line bb_repo_urls = [], but create a empty list later and append directly in while loop?
Any way to improve this line : "".join(map(lambda url...

Question 2

One suggestion: perhaps consider using the keyring module rather than storing Bitbucket credentials in settings.py.

Question 3

Its actually very simple, there is no need to write complex scripts.

What I did was, complete one import process using BitBuckets HTML form, and record the post request that form makes using chrome. The chrome will give you the request in cURL form. Then use a simple nodejs script to run the same cURL command by just replacing the repo URL and the repo name.

var shell = require('shelljs');
var arr = {
 "https://laxxxxxxtee.git": "pyyyyye",
 "https://lxxxxxxxxlace.git": "shyyyyyce",
 "https://latttttttwa.git": "syyyyya",
 "https://laxxxxxxxappy-ionic.git": "snayyyyic",
 "https://laxxxxxxxxxxxxnter-ui.git": "viyyyyyyr-ui",
 "https://laxxxxxxxxxnter.git": "vyyyyyer",
 "https://laxxxxxxxxtend.git": "viyyyyy"
};
for (var url in arr) {
 var name = arr[url];
 name = encodeURIComponent(name);
 url = encodeURIComponent(url);
 var command = "curl 'https://bitbucket.org/repo/import?owner=txxxxh' -H 'Pragma: no-cache' -H 'Origin: https://bitbucket.org' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8,en-GB;q=0.6,ml;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://bitbucket.org/repo/import?owner=txxxxxh' -H 'Cookie: optimizelyEndUserId=oexxxxxxxxxx643798; __ar_v4=3coookies-form' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'source_scm=git&source=source-git&goog_project_name=&goog_scm=svn&sourceforge_project_name=&sourceforge_mount_point=&sourceforge_scm=svn&codeplex_project_name=&codeplex_scm=svn&url=" + url + ".git&auth=on&username=nyyyyyys&password=Gvvvvv&owner=36vvvv2&name=" + name + "&description=&is_private=on&forking=no_forks&no_forks=True&no_public_forks=True&has_wiki=on&language=php&csrfmiddlewaretoken=MErrrrrrrrrOZ91r&scm=git' --compressed";
 shell.exec(command, function(code, output) {
 console.log('Exit code:', code);
 console.log('Program output:', output);
 });
}

Question 4

Thanks! Minimal curl with basic auth:

curl 'https://bitbucket.org/repo/import' -H 'Content-Type: application/x-www-form-urlencoded' --data 'source_scm=git&source=source-git&sourceforge_scm=hg&codeplex_scm=hg&forking=no_public_forks&no_forks=False&no_public_forks=True&scm=git&url=<URL-encoded GitHub repo URL>&is_private=on&username=<GitHub username>&password=<GitHub password>&owner=<BitBucket user ID>&name=<BitBucket repo name>' -H 'Authorization: <basic auth>'

BitBucket user ID: "owner" field on import form, or "id" field on "bb-bootstrap" HTML meta tag on any page

nidheeshdas nidheeshdas 1511 silver badge2 bronze badges · Answer 1 · 2014-12-03 17:51:40Z

Its actually very simple, there is no need to write complex scripts.

What I did was, complete one import process using BitBuckets HTML form, and record the post request that form makes using chrome. The chrome will give you the request in cURL form. Then use a simple nodejs script to run the same cURL command by just replacing the repo URL and the repo name.

var shell = require('shelljs');
var arr = {
 "https://laxxxxxxtee.git": "pyyyyye",
 "https://lxxxxxxxxlace.git": "shyyyyyce",
 "https://latttttttwa.git": "syyyyya",
 "https://laxxxxxxxappy-ionic.git": "snayyyyic",
 "https://laxxxxxxxxxxxxnter-ui.git": "viyyyyyyr-ui",
 "https://laxxxxxxxxxnter.git": "vyyyyyer",
 "https://laxxxxxxxxtend.git": "viyyyyy"
};
for (var url in arr) {
 var name = arr[url];
 name = encodeURIComponent(name);
 url = encodeURIComponent(url);
 var command = "curl 'https://bitbucket.org/repo/import?owner=txxxxh' -H 'Pragma: no-cache' -H 'Origin: https://bitbucket.org' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8,en-GB;q=0.6,ml;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Cache-Control: no-cache' -H 'Referer: https://bitbucket.org/repo/import?owner=txxxxxh' -H 'Cookie: optimizelyEndUserId=oexxxxxxxxxx643798; __ar_v4=3coookies-form' -H 'Connection: keep-alive' -H 'DNT: 1' --data 'source_scm=git&source=source-git&goog_project_name=&goog_scm=svn&sourceforge_project_name=&sourceforge_mount_point=&sourceforge_scm=svn&codeplex_project_name=&codeplex_scm=svn&url=" + url + ".git&auth=on&username=nyyyyyys&password=Gvvvvv&owner=36vvvv2&name=" + name + "&description=&is_private=on&forking=no_forks&no_forks=True&no_public_forks=True&has_wiki=on&language=php&csrfmiddlewaretoken=MErrrrrrrrrOZ91r&scm=git' --compressed";
 shell.exec(command, function(code, output) {
 console.log('Exit code:', code);
 console.log('Program output:', output);
 });
}

Thanks! Minimal curl with basic auth: curl 'https://bitbucket.org/repo/import' -H 'Content-Type: application/x-www-form-urlencoded' --data 'source_scm=git&source=source-git&sourceforge_scm=hg&codeplex_scm=hg&forking=no_public_forks&no_forks=False&no_public_forks=True&scm=git&url=<URL-encoded GitHub repo URL>&is_private=on&username=<GitHub username>&password=<GitHub password>&owner=<BitBucket user ID>&name=<BitBucket repo name>' -H 'Authorization: <basic auth>' BitBucket user ID: "owner" field on import form, or "id" field on "bb-bootstrap" HTML meta tag on any page

Stack Exchange Network

Script to import repositories to Bitbucket

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Hot Network Questions

Script to import repositories to Bitbucket

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Related

Hot Network Questions