people_office_id: snip: <class 'lxml.etree._ElementUnicodeResult'>
people_office_id: snip: /organisation/people/387021/t-g-baynes-solicitors?Solicitors=True
people_office_id: snip.strip(): /organisation/people/387021/t-g-baynes-solicitors?Solicitors=True
387021
organisation/people
apply_xpath: results xpath: string(.//*[@id='main-details-accordion']/dl[2]/dd[2])
apply_xpath: results type: <class 'lxml.etree._ElementUnicodeResult'>
apply_xpath: results:  DX 151360 Bexleyheath 6
apply_xpath: results xpath: .//*[@id='main-details-accordion']/div/p/img/@data-original-title
apply_xpath: results type: <class 'list'>
apply_xpath: results:  []
[]
-- Company Name: Braund & Fedrick
-- Telephone Number: 02083006515
-- Website URL: 
-- Email: 
-- sraID: 598889
-- Solicitor ID: 546940
-- Long: 51.42447
-- Lat: 0.1078216
-- Solicitor Type: Recognised body law practice
-- Address: 
['13 Market Parade', ' High Street', ' Sidcup', ' Kent', ' DA14 6EP', ' England']
-- Tel: 02083006515
-- DX Address: DX 151360 Bexleyheath 6
<class 'str'>
-- Facilities: []
person_loop: url: http://solicitors.lawsociety.org.uk/organisation/people/387021/?Solicitors=True
fetch_url_requests: Sending request with requests...
fetch_url_requests: url: http://solicitors.lawsociety.org.uk/organisation/people/387021/?Solicitors=True
fetch_url_requests: text: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<title>Proxy error: 502 Disconnected operation and object not in cache.</title>
</head><body>
<h1>502 Disconnected operation and object not in cache</h1>
<p>The following error occurred while trying to access <strong>http://solicitors.lawsociety.org.uk/organisation/people/387021/t-g-baynes-solicitors</strong>:<br><br>
<strong>502 Disconnected operation and object not in cache</strong></p>
<hr>Generated Tue, 01 Sep 2015 03:15:09 BST by Polipo on <em>sparky:8123</em>.
</body></html>

apply_xpath: results xpath: .//*[@class='search-results-controls']//h1/strong/text()
apply_xpath: results type: <class 'list'>
apply_xpath: results:  []
Traceback (most recent call last):
  File "test_alphabet2.py", line 859, in <module>
    main(max_results, name)
  File "test_alphabet2.py", line 802, in main
    office_loop(results, prefix)
  File "test_alphabet2.py", line 667, in office_loop
    output_office_details(url, results_tree, idnumber)
  File "test_alphabet2.py", line 572, in output_office_details
    person_loop(people_url)
  File "test_alphabet2.py", line 627, in person_loop
    print("person_loop: results: str(results[0]): " + str(results[0]))
IndexError: list index out of range

http://solicitors.lawsociety.org.uk/office/546940/braund-fedrick
http://solicitors.lawsociety.org.uk/office/546940/braund-fedrick#people-and-structure-accordion

http://solicitors.lawsociety.org.uk/office/387021/t-g-baynes-solicitors
http://solicitors.lawsociety.org.uk/office/people/387021/t-g-baynes-solicitors?Solicitors=True

--------------------
person_loop: url: http://solicitors.lawsociety.org.uk/organisation/people/543151/?Solicitors=True
fetch_url_requests: Sending request with requests...
fetch_url_requests: url: http://solicitors.lawsociety.org.uk/organisation/people/543151/?Solicitors=True
fetch_url_requests: text: <h2>Server Too Busy</h2>

apply_xpath: results xpath: .//*[@class='search-results-controls']//h1/strong/text()
apply_xpath: results type: <class 'list'>
apply_xpath: results:  []
Traceback (most recent call last):
  File "test_alphabet2.py", line 871, in <module>
    main(max_results, name)
  File "test_alphabet2.py", line 792, in main
    office_loop(results, prefix)
  File "test_alphabet2.py", line 679, in office_loop
    output_office_details(url, results_tree, idnumber)
  File "test_alphabet2.py", line 584, in output_office_details
    person_loop(people_url)
  File "test_alphabet2.py", line 639, in person_loop
    print("person_loop: results: str(results[0]): " + str(results[0]))
IndexError: list index out of range

--------------------
polipo_off: Turning Polipo off.
------------------------------------------------------------------------------------------------------
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Polipo response.status_code: 200
pk
get_numberOfResults: name: pk
get_numberOfResults: page: 1
Lawsoc results page URL already saved.
fetch_url_requests: Sending request with requests...
fetch_url_requests: url: http://solicitors.lawsociety.org.uk/search/results?Name=pk&Type=0&IncludeNlsp=false&Pro=True&Page=1
fetch_url_requests: text: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<title>Proxy error: 502 Disconnected operation and object not in cache.</title>
</head><body>
<h1>502 Disconnected operation and object not in cache</h1>
<p>The following error occurred while trying to access <strong>http://solicitors.lawsociety.org.uk/search/results?Name=pk&amp;Type=0&amp;IncludeNlsp=false&amp;Pro=True&amp;Page=1</strong>:<br><br>
<strong>502 Disconnected operation and object not in cache</strong></p>
<hr>Generated Fri, 04 Sep 2015 12:50:57 BST by Polipo on <em>sparky:8123</em>.
</body></html>

apply_xpath: results xpath: string(.//*[@id='main-content']//h1[contains(.,'Found')]/strong[1])
apply_xpath: results type: <class 'lxml.etree._ElementUnicodeResult'>
apply_xpath: results:  
Traceback (most recent call last):
  File "test_alphabet2.py", line 876, in <module>
    main(max_results, name)
  File "test_alphabet2.py", line 791, in main
    results = get_numberOfResults(prefix, 1)
  File "test_alphabet2.py", line 171, in get_numberOfResults
    print("get_numberOfResults: Value of results before casting to int: " + str(results[0]))
IndexError: string index out of range

--------------------
dropped connection - wifi


+++ Solicitor Name: Sonita Devi Hayward
+++ Admitted: 01/10/08
+++ sraID: 423960
+++ Office Name: Bolt Burdon Kemp
+++ Person ID: 73722
+++ Solicitor ID: 403724
+++ DX Address: 
+++ Role: Associate
+++ Email: info@boltburdonkemp.co.uk
+++ Tel: 020 7288 4800
+++ Accreditations: []
+++ Areas of Practice: ['Litigation - general', 'Commercial litigation', 'Professional negligence']
+++ Languages: ['Creole', 'English', 'French']
person_loop: person_url: page_urls: /person/275248/ruth-hewitt
URL already fetched, next.
fetch_url_requests: Sending request with requests...
fetch_url_requests: url: http://solicitors.lawsociety.org.uk/person/275248/
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 331, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 333, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.4/http/client.py", line 1172, in getresponse
    response.begin()
  File "/usr/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.4/http/client.py", line 321, in _read_status
    raise BadStatusLine(line)
http.client.BadStatusLine: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 559, in urlopen
    _pool=self, _stacktrace=stacktrace)
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 245, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 624, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 333, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.4/http/client.py", line 1172, in getresponse
    response.begin()
  File "/usr/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.4/http/client.py", line 321, in _read_status
    raise BadStatusLine(line)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BadStatusLine("''",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_alphabet2.py", line 859, in <module>
    main(max_results, name)
  File "test_alphabet2.py", line 780, in main
    office_loop(results, prefix)
  File "test_alphabet2.py", line 667, in office_loop
    output_office_details(url, results_tree, idnumber)
  File "test_alphabet2.py", line 512, in output_office_details
    output_office_details(url, results_tree, idnumber)
  File "test_alphabet2.py", line 572, in output_office_details
    person_loop(people_url)
  File "test_alphabet2.py", line 648, in person_loop
    output_person_details(url)
  File "test_alphabet2.py", line 438, in output_person_details
    person_tree = make_simple_tree(url)
  File "test_alphabet2.py", line 137, in make_simple_tree
    text = fetch_url(url)
  File "test_alphabet2.py", line 27, in fetch_url
    text = fetch_url_requests(url)
  File "test_alphabet2.py", line 43, in fetch_url_requests
    r = requests.get(url, proxies=polipo_proxies)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 595, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 595, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 189, in resolve_redirects
    allow_redirects=False,
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 407, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

