API

CAUTION: Using API during a running crawl may be unresponsive at times and slow down the crawl because it may occasionally activate creation of optimized tables for certain queries, which take more time for bigger crawls. It is best to wait for the crawl to finish, and then use the API.

API URL is of the folowing form:

http://{IP}:{PORT}/u/?id={JOB_ID}&a={ACTION_ID}&h={JOB_HASH_CODE}[&q1={QUERY_STRING_1}][&q2={QUERY_STRING_2}][&l={LIMIT_RESULTS}][&o={OFFSET}]

IP, PORT, JOB_ID and JOB_HASH_CODE are provided at the job's dashboard page upon job creation. Issuing DELETE RESULTS then START may assign a new IP. Depending on the ACTION_ID issued, QUERY_STRING_1 and QUERY_STRING_2 may be mandatory or not. OFFSET and LIMIT_RESULTS are always optional.

parameter	description	mandatory or optional	type
a	ACTION_ID	mandatory	integer
id	JOB_ID	mandatory	integer
q1	QUERY_STRING_1	mandatory for some actions	URL encoded string (URL or query phrase)
q2	QUERY_STRING_2	mandatory for some actions	URL encoded string (URL or query phrase)
l	LIMIT_RESULTS	optional	integer (1000 default & maximum)
o	OFFSET	optional for some actions	integer
h	JOB_HASH_CODE	mandatory	string

^{[] placing QUERY_STRING_1 within square brackets [] in some commands results in exact query match}

If OFFSET is specified, then instead of latest results, results starting with specified OFFSET id will be retrieved. This way, user can navigate the whole set of results through the API.

Response is JSON formatted output. Every command having 'Page' or 'Title' in name, if it does not return 'on_topic' field, all results are by default on_topic! Pages table contains only on-topic pages, so every query over that table by default includes only on-topic results. Queries that include edges as well contain on_topic field for the target/source node.

ACTION_ID	description	mandatory parameters	optional parameters
83	most relevant domains
80	domains with most linking domains
79	domains with most incoming links
82	domains with most domains linked
81	domains with most outgoing links
121	tld statistics
24	keyword ideas
118	keywords from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
119	atext keywords to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
120	atext keywords from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
21	atext
76	atext to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
23	atext to http://www.example.com/	&q1=http%3A%2F%2Fwww.example.com%2F
43	query phrase in atext	&q1=query+phrase
98	query phrase in atext to example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F
47	query phrase in atext to http://www.example.com/	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F
97	query phrase in atext from example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F
48	query phrase in title	&q1=query+phrase
99	query phrase in title from example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F
37	most linked pages
90	most linked pages from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
28	hubs
84	hubs from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
50	popular pages
114	popular prospect pages
104	popular pages pointing to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
59	popular pages pointing to http://www.example.com/	&q1=http%3A%2F%2Fwww.example.com%2F
103	popular pages from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
117	popular prospect pages from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
51	popular pages linking with query phrase in atext	&q1=query+phrase	[]
101	popular pages linking to example.com with query phrase in atext	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
54	popular pages linking to http://www.example.com/ with query phrase in atext	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
100	popular pages from example.com linking with query phrase in atext	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
55	popular pages with query phrase in title	&q1=query+phrase	[]
115	popular prospect pages with query phrase in title	&q1=query+phrase	[]
102	popular pages with query phrase in title from example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
116	popular prospect pages with query phrase in title from example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
60	urls with query phrase in atext	&q1=query+phrase	[]
106	urls with query phrase in atext to example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
64	urls with query phrase in atext to http://www.example.com/	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
105	urls with query phrase in atext from example.com	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
62	urls with query phrase in atext from http://www.example.com/	&q1=query+phrase&q2=http%3A%2F%2Fwww.example.com%2F	[]
65	urls with query phrase in title	&q1=query+phrase
78	domains with most semantic flow to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
77	domains with most semantic flow from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
122	tld semantic flows to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
123	tld semantic flows from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
91	number inbound links to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
92	number inbound domains to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
93	number outbound links from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
94	number outbound domains from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
108	example.com rank	&q1=http%3A%2F%2Fwww.example.com%2F
107	similar domains to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
88	links to example.com	&q1=http%3A%2F%2Fwww.example.com%2F
36	links to http://www.example.com/	&q1=http%3A%2F%2Fwww.example.com%2F
95	links from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
41	links from http://www.example.com/	&q1=http%3A%2F%2Fwww.example.com%2F
96	pages from example.com	&q1=http%3A%2F%2Fwww.example.com%2F
89	links to example.com from source.com	&q1=http%3A%2F%2Fwww.example.com%2F&q2=http%3A%2F%2Fsource.com%2F
109	links from source.com to example.com	&q1=http%3A%2F%2Fsource.com%2F&q2=http%3A%2F%2Fwww.example.com%2F
3	latest pages		&o=12345
4	latest URLs		&o=12345
5	latest edges		&o=12345
16	page content for http://www.example.com/	&q1=http%3A%2F%2Fwww.example.com%2F
10	crawl download links

Note that PR is meaningless in 'seed only' crawls and "popular pages" actions are therefore useless in those cases. For regular 'unlimited' crawls however, these are very useful commands. Also, for single site crawls it is useful to analyze popular pages to see website's internal linking effects.