Syncto¶
Syncto is a server allowing you to store and retrieve Firefox Sync user data attached to your Firefox account using a subset of the Kinto API in order to be able to use Kinto.js for that task.
Table of content¶
Overview¶
Syncto is a bridge service that let you interract with the Firefox Sync infrastructure using Kinto clients such as Kinto.js or Kinto.py.
More information¶
FAQ¶
Is Syncto secure?¶
Syncto caches encrypted credentials for five minutes for a given BrowserID assertion.
In addition all the data coming from Firefox Sync are encrypted and Syncto will never have access to the encryption key (only final clients have).
By default Syncto is a read-only proxy, to enable write permissions for some collection, it has to be configured explicitely. (history/password/tabs/etc.)
Is it web scale?¶
Syncto is simply a proxy that relies on the Firefox Sync infrastructure.
Syncto doesn’t even necessarily need a shared cache between its nodes.
Also it is better that nodes from the same loadbalancer uses the same cache or that client are always affected to the same one with regards to their Authorization header.
What is Cliquet? What is the difference between Cliquet, Syncto and Kinto ?¶
Cliquet is a toolkit for designing micro-services. Kinto and Syncto are servers built using that toolkit.
Syncto uses the same protocol as Kinto does to name and interract with collection records. This enable developers to use Kinto clients to fetch Firefox Sync collection records.
I am seeing an Exception error, what’s wrong?¶
Have a look at the Troubleshooting section to see what to do.
Installation¶
Run locally¶
Syncto is based on top of the cliquet project, and as such, you may wanto to refer to cliquet’s documentation for more details.
For development¶
By default, Syncto persists internal cache in Redis.
git clone https://github.com/mozilla-services/syncto
cd syncto
make serve
note: | OSX users are warned that supplementary steps are needed to ensure proper installation of cryptographic dependencies is properly done; see dedicated note. |
---|
If you already installed Syncto earlier and you want to recreate a
full environment (because of errors when running make serve
), please run:
make maintainer-clean serve
Authentication¶
Syncto relies on Firefox Account BrowserID assertion to authenticate users to the Token Server.
Note that you will need to pass through a BrowserID assertion with the https://token.services.mozilla.com/ audience for Syncto to be able to read Firefox Sync server credentials.
This can be made using HTTPie and PyFxA.
To install them:
$ pip install httpie PyFxA
To build a Firefox Account Browser ID assertion for an existing user:
$ BID_AUDIENCE=https://token.services.mozilla.com/ \
BID_WITH_CLIENT_STATE=True \
http GET 'https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records'
--auth-type fxa-browserid --auth "user@email.com:US3R_P4S5W0RD" -v
Once you have got a BrowserID Assertion, you can also reuse it to benefit from Syncto cache features:
$ http GET 'https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records' \
Authorization:"BrowserID eyJhbGciOiJSUzI1NiJ9..." \
X-Client-State:64e8bc35e90806f9a67c0ef8fef63...
Cryptography libraries¶
Linux¶
On Debian / Ubuntu based systems:
apt-get install libffi-dev libssl-dev
On RHEL-derivatives:
apt-get install libffi-devel openssl-devel
OS X¶
Assuming brew is installed:
brew install libffi openssl pkg-config
Warning
Apple having dropped support for OpenSSL and moving to their own library recently, you have to force its usage to properly install cryptography-related dependencies:
$ env LDFLAGS="-L$(brew --prefix openssl)/lib" \
CFLAGS="-I$(brew --prefix openssl)/include" \
.venv/bin/pip install cryptography
$ make serve
Running in production¶
Recommended settings¶
Most default setting values in the application code base are suitable for production.
However, the set of settings mentionned below might deserve some review or adjustments:
syncto.cache_backend = cliquet.cache.redis
syncto.cache_url = redis://localhost:6379/1
syncto.http_scheme = https
syncto.http_host = <hostname>
syncto.retry_after_seconds = 30
syncto.batch_max_requests = 25
syncto.cache_hmac_secret = <32 random bytes as hex>
syncto.token_server_url = https://token.services.mozilla.com/
note: | For an exhaustive list of available settings and their default values, refer to cliquet source code.
|
---|
Enable write access¶
By default, collections are read-only. In order to enable write operations on remote Sync collections, add some settings in the configuration with the collection name:
syncto.record_tabs_put_enabled = true
syncto.record_tabs_delete_enabled = true
syncto.record_passwords_put_enabled = true
syncto.record_passwords_delete_enabled = true
syncto.record_bookmarks_put_enabled = true
syncto.record_bookmarks_delete_enabled = true
syncto.record_history_put_enabled = true
syncto.record_history_delete_enabled = true
Monitoring¶
# Heka
syncto.logging_renderer = cliquet.logs.MozillaHekaRenderer
# StatsD
syncto.statsd_url = udp://carbon.server:8125
Application output should go to stdout
, and message format should have no
prefix string:
[handler_console]
class = StreamHandler
args = (sys.stdout,)
level = INFO
formater = heka
[formatter_heka]
format = %(message)s
Adapt the logging configuration in order to plug Sentry:
[loggers]
keys = root, sentry
[handlers]
keys = console, sentry
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console, sentry
[logger_sentry]
level = WARN
handlers = console
qualname = sentry.errors
propagate = 0
[handler_console]
class = StreamHandler
args = (sys.stdout,)
level = INFO
formater = heka
[formatter_heka]
format = %(message)s
[handler_sentry]
class = raven.handlers.logging.SentryHandler
args = ('http://public:secret@example.com/1',)
level = WARNING
formatter = generic
[formatter_generic]
format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
Running with uWsgi¶
To run the application using uWsgi, an app.wsgi file is provided. This command can be used to run it:
uwsgi --ini config/syncto.ini
uWsgi configuration can be tweaked in the ini file in the dedicated [uwsgi] section.
Here’s an example:
[uwsgi]
wsgi-file = app.wsgi
enable-threads = true
socket = /run/uwsgi/syncto.sock
chmod-socket = 666
cheaper-algo = busyness
cheaper = 5
cheaper-initial = 9
workers = 14
cheaper-step = 1
cheaper-overload = 30
cheaper-busyness-verbose = true
master = true
module = syncto
harakiri = 120
uid = ubuntu
gid = ubuntu
virtualenv = /data/venvs/syncto
lazy = true
lazy-apps = true
single-interpreter = true
buffer-size = 65535
post-buffering = 65535
To use a different ini file, the SYNCTO_INI
environment variable
should be present with a path set to it.
API Endpoints¶
Resource endpoints¶
Get a list of records for a collection¶
Requires authentication
Returns all records of the current user for this collection.
Collection can be one of the Firefox Sync collections:
The default collections used by Firefox to store sync data are:
- bookmarks
- history
- forms
- prefs
- tabs
- passwords
The following additional collections are used for internal management purposes by the storage client:
- clients
- crypto
- keys
- meta
The returned value is a JSON mapping containing:
data
: the list of records, with exhaustive fields;
A Total-Records
response header indicates the total number of records
in the collection.
A Last-Modified
response header provides a human-readable (rounded
to second) version of the current collection timestamp.
For cache and concurrency control, an ETag
response header gives the
value that consumers can provide in subsequent requests using If-Match
and If-None-Match
headers (see the section about timestamps).
-
GET
/buckets/syncto/collections/
(collection_id)/records
¶ Example request:
$ BID_AUDIENCE=https://token.services.mozilla.com/ BID_WITH_CLIENT_STATE=True \ http GET https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records \ --auth-type fxa-browserid --auth "user@email.com:P4S5W0RD" -v GET /v1/buckets/syncto/collections/history/records HTTP/1.1 Authorization: BrowserID eyJhbGciOiJSUzI1NiJ9...FHGg Host: syncto.dev.mozaws.net User-Agent: HTTPie/0.9.2 X-Client-State: 64e8bc35e90806f9a67c0ef8fef63...
Example response:
HTTP/1.1 200 OK Access-Control-Expose-Headers: Content-Length, Quota-Remaining, Alert, \ Retry-After, Last-Modified, Total-Records, ETag, Backoff, Next-Page Content-Length: 1680 Content-Type: application/json; charset=UTF-8 Date: Tue, 06 Oct 2015 13:57:24 GMT ETag: "1442849064460" Last-Modified: Mon, 21 Sep 2015 15:24:24 GMT Total-Records: 6 { "data": [ { "id": "VLkOS7iT5C94", "last_modified": 1441868927070, "payload": "{\"ciphertext\":\"Wf2AoZiOly...\",\"IV\":\"jW7JFPf...\",\"hmac\":\"989352d9b5e0c6...\"}", "sortindex": -1 }, { "id": "qYYobAN_p9vS", "last_modified": 1441868927070, "payload": "{\"ciphertext\":\"3upjoLrO...7\",\"IV\":\"3O/nPq82xUT...\",\"hmac\":\"addce0f9d3024ed9fd0042b...\"}", "sortindex": 100 }, ... ] }
Status Codes: - 200 OK – The request was processed.
- 304 Not Modified – The collection did not change since value in
If-None-Match
header - 400 Bad Request – The request querystring is invalid
- 401 Unauthorized – Something went wrong with your authentication
- 412 Precondition Failed – Collection changed since value in
If-Match
header
Filtering and Sorting¶
Firefox Sync filtering options are exposed in syncto.
_since
with the ETag value to fetch changes from the previous time._sort
can be eithernewest
,oldest
, orindex
(as well as-last_modified
,last_modified
, and-sortindex
)._limit
to limit the number of items per pages (no limit by default).in_ids
to define the list of requested records IDs.
Pagination¶
The Next-Page
header will be sent with the URL to fetch the next
page. It will include defined _limit
and _token
values
automatically.
When the Next-Page
is not present, it means there is no more data
to fetch.
Counting¶
Contrary to what Kinto does, the Total-Records
only counts the
number of records contained in the current request
for now.
You may ask the request without the _limit
parameter to get all
the records at once.
Polling for changes¶
The _since
parameter is provided as an alias for gt_last_modified
.
(Greater than last_modified
)
If the request header If-None-Match
is provided as described in
the section about timestamps and if the
collection was not changed, a 304 Not Modified
response is returned.
Additionnal headers¶
The Quota-Remaining
header is not part of the Kinto protocol yet
but is passed through if present in Firefox Sync responses.
Its value is in Kilobyte (KB).
Get a collection record¶
Requires authentication
Returns a specific record by its id. The GET response body is a JSON mapping containing:
data
: the record with exhaustive schema fields;
IDs are kept between Firefox Sync and Syncto.
Firefox Sync IDs are generated on client side as 9 random Bytes
encoded in urlsafe base64 (+
and /
are replaced with -
and
_
).
If the request header If-None-Match
is provided, and if the record has not
changed meanwhile, a 304 Not Modified
is returned.
-
GET
/buckets/syncto/collections/
(collection_id)/records/
(record_id)¶ Example request:
$ http GET \ https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS \ Authorization:"BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ" \ X-Client-State:64e8bc35e90806f9a67c0ef8fef63... GET /v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS HTTP/1.1 Authorization: BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ Host: syncto.dev.mozaws.net User-Agent: HTTPie/0.9.2 X-Client-State: 64e8bc35e90806f9a67c0ef8fef63...
Example response:
HTTP/1.1 200 OK Access-Control-Expose-Headers: Content-Length, Alert, Retry-After, Last-Modified, ETag, Backoff Content-Length: 289 Content-Type: application/json; charset=UTF-8 Date: Tue, 06 Oct 2015 14:18:40 GMT ETag: "1441868927070" Last-Modified: Thu, 10 Sep 2015 07:08:47 GMT { "data": { "id": "d2X1O6-DyeFS", "last_modified": 1441868927070, "payload": "{\"ciphertext\":\"75IcW3P4WxUJipehWryevc+ygK5vojh3n...\",\"IV\":\"Sj3U2Nkk2IjE...\",\"hmac\":\"c6a530f348...b68b610351\"}", "sortindex": 2000 } }
Status Codes: - 200 OK – The request was processed.
- 304 Not Modified – The collection did not change since value in
If-None-Match
header - 401 Unauthorized – Something went wrong with your authentication
- 412 Precondition Failed – Collection changed since value in
If-Match
header
Delete a record¶
Requires authentication
Delete a specific record by its id.
Note that contrary to what Kinto does, Firefox Sync count on clients to create deleted records tombstones. Moreover Firefox Sync tombstones are encrypted and look like real records for Syncto.
This endpoint should not be used to create tombstones but to remove the record when the client decides that all clients already fetched the tombstone.
By default this endpoint is deactivated and should be activated on a per collection basis.
-
DELETE
/buckets/syncto/collections/
(collection_id)/records/
(record_id)¶ Example request:
$ http DELETE \ https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS \ Authorization:"BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ" \ X-Client-State:64e8bc35e90806f9a67c0ef8fef63... DELETE /v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS HTTP/1.1 Authorization: BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ Host: syncto.dev.mozaws.net User-Agent: HTTPie/0.9.2 X-Client-State: 64e8bc35e90806f9a67c0ef8fef63...
Example response:
HTTP/1.1 204 No Content Access-Control-Expose-Headers: Content-Length, Alert, Retry-After, Last-Modified, ETag, Backoff Content-Length: 0 Date: Tue, 06 Oct 2015 14:18:40 GMT
Status Codes: - 204 No Content – The record was deleted
- 401 Unauthorized – Something went wrong with your authentication
- 405 Method Not Allowed – This endpoint was not activated in the configuration
- 412 Precondition Failed – Collection changed since value in
If-Match
header
Create or Update a record¶
Requires authentication
Create or replace a record with its id. The PUT body is a JSON mapping containing:
data
: the values of the resource schema fields;
Because IDs are created on client side for Firefox Sync, this is the only endpoint that you can use either to create new record or to update them.
If you want to make sure that you don’t erase an existing record when
creating one, you can use the If-None-Match: *
header value.
The PUT response body is a JSON mapping containing:
data
: the newly created/updated record, if all posted values are valid;
If the request header If-Match
is provided, and if the record has
changed meanwhile, a 412 Precondition failed
error is returned.
There are no validation nor on the id format nor on the payload body.
By default this endpoint is deactivated and should be activated on a per collection basis.
-
PUT
/buckets/syncto/collections/
(collection_id)/records/
(record_id)¶ Example request:
$ echo '{ "data": { "payload": "{\"ciphertext\":\"75IcW3P4WxUJipehWryevc+ygK5vojh3nOadu7YSX9zJSm3eBHu5lNIg1UtDyt3b\",\"IV\":\"Sj3U2Nkk2IjE2S59hv0m7Q==\",\"hmac\":\"c6a530f3486142d1069f80bfaff907e0cc077a892cf7f9bd62f943b68b610351\"}", "sortindex": 2000 } }' | http PUT \ https://syncto.dev.mozaws.net/v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS \ Authorization:"BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ" \ X-Client-State:64e8bc35e90806f9a67c0ef8fef63... PUT /v1/buckets/syncto/collections/history/records/d2X1O6-DyeFS HTTP/1.1 Authorization: BrowserID eyJhbGciOiJSUzI1NiJ9...i_dQ Content-Length: 275 Content-Type: application/json Host: syncto.dev.mozaws.net User-Agent: HTTPie/0.9.2 X-Client-State: 64e8bc35e90806f9a67c0ef8fef63... { "data": { "payload": "{\"ciphertext\":\"75IcW3P4WxUJipehWryevc+ygK5vojh3nOadu7YSX9zJSm3eBHu5lNIg1UtDyt3b\",\"IV\":\"Sj3U2Nkk2IjE2S59hv0m7Q==\",\"hmac\":\"c6a530f3486142d1069f80bfaff907e0cc077a892cf7f9bd62f943b68b610351\"}", "sortindex": 2000 } }
Example response:
HTTP/1.1 200 OK Access-Control-Expose-Headers: Content-Length, Alert, Retry-After, Last-Modified, ETag, Backoff Connection: keep-alive Content-Length: 289 Content-Type: application/json; charset=UTF-8 Date: Fri, 09 Oct 2015 10:04:13 GMT ETag: "1444385059190" Last-Modified: Fri, 09 Oct 2015 10:04:19 GMT { "data": { "id": "d2X1O6-DyeFS", "last_modified": 1444385059190, "payload": "{\"ciphertext\":\"75IcW3P4WxUJipehWryevc+ygK5vojh3nOadu7YSX9zJSm3eBHu5lNIg1UtDyt3b\",\"IV\":\"Sj3U2Nkk2IjE2S59hv0m7Q==\",\"hmac\":\"c6a530f3486142d1069f80bfaff907e0cc077a892cf7f9bd62f943b68b610351\"}", "sortindex": 2000 } }
Status Codes: - 200 OK – The record was created or updated
- 400 Bad Request – The request body is invalid
- 401 Unauthorized – Something went wrong with your authentication
- 405 Method Not Allowed – This endpoint was not activated in the configuration
- 412 Precondition Failed – Collection changed since value in
If-Match
header
Utility endpoints for OPS and Devs¶
GET /¶
The returned value is a JSON mapping containing:
Changed in version 2.12.
project_name
: the name of the service (e.g."reading list"
)project_docs
: The URL to the service documentation. (this document!)project_version
: complete application/project version ("3.14.116"
)http_api_version
: the MAJOR.MINOR version of the exposed HTTP API ("1.1"
) defined in configuration.cliquet_protocol_version
: the cliquet protocol version ("2"
)url
: absolute URI (without a trailing slash) of the API (can be used by client to build URIs)eos
: date of end of support in ISO 8601 format ("yyyy-mm-dd"
, undefined if unknown)settings
: a mapping with the values of relevant public settings for clientsbatch_max_requests
: Number of requests that can be made in a batch request.readonly
: Only requests with read operations are allowed.
capabilities
: a mapping used by clients to detect optional features of the API.Example:
{ "auth-fxa": { "description": "Firefox Account authentication", "url": "http://github.com/mozilla-services/cliquet-fxa" } }
Optional
user
: A mapping with anid
field for the currently connected user id. The field is not present when no Authorization header is provided.
Note
The project_version
contains the source code version, whereas the http_api_version
contains the exposed HTTP API version.
The source code of the service can suffer changes and have its project version incremented, without impacting the publicly exposed HTTP API.
The cliquet_protocol_version
is an internal notion tracking the version
for some aspects of the API (e.g. synchronization of REST resources, utilities endpoints, etc.). It will differ from the http_api_version
since the service
will provide additionnal endpoints and conventions.
GET /__heartbeat__¶
Return the status of each service the application depends on. The returned value is a JSON mapping containing:
storage
true if storage backend is operationalcache
true if cache backend operationalpermission
true if permission backend operational
If cliquet-fxa
is installed, an additional key is present:
oauth
true if authentication is operational
Return 200
if the connection with each service is working properly
and 503
if something doesn’t work.
GET /__lbheartbeat__¶
Always return 200
with empty body.
Unlike the __heartbeat__
health check endpoint, which return an error
when backends and other upstream services are unavailable, this should
always return 200.
This endpoint is suitable for a load balancer membership test. It the load balancer cannot obtain a response from this endpoint, it will stop sending traffic to the instance and replace it.
API Versioning¶
The HTTP API exposed by the service will be consumed by clients, like a Javascript client.
The HTTP API is subject to changes. It follows the Cliquet Protocol.
When the HTTP API is changed, its version is incremented. The HTTP API version follows a Semantic Versioning pattern and uses this rule to be incremented:
- any change to the HTTP API that is backward compatible increments the MINOR number, and the modification in the documentation should reflect this with a header like “Added in 1.x”.
- any change to the HTTP API that is backward incompatible increments the MAJOR number, and the differences are summarized at the begining of the documentation, a new document for that MAJOR version is created.
Note
We’re not using the PATCH level of Semantic Versioning, since bug fixes have no impact on the exposed HTTP API; if they do MINOR or MAJOR should be incremented.
We want to avoid MAJOR changes as much as possible in the future, and stick with 1.x as long as we can.
A client that interacts with the service can query the server to know what is its HTTP API version. This is done with a query on the root view, as described in the root API description.
If a client relies on a feature that was introduced at a particular version, it should check that the server implements the minimal required version.
The JSON response body contains an http_api_version key which value is the MAJOR.MINOR version.
Batch operations¶
POST /batch¶
Requires authentication
The POST body is a mapping, with the following attributes:
requests
: the list of requestsdefaults
: (optional) default requests values in common for all requests
Each request is a JSON mapping, with the following attribute:
method
: HTTP verbpath
: URIbody
: a mappingheaders
: (optional), otherwise take those of batch request
POST /batch HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Content-Length: 728
Host: localhost:8888
User-Agent: HTTPie/0.9.2
{
"defaults": {
"method" : "POST",
"path" : "/articles",
},
"requests": [
{
"body" : {
"data" : {
"title": "MoFo",
"url" : "http://mozilla.org",
"added_by": "FxOS",
},
"permissions": {
"read": ["system.Everyone"]
}
}
},
{
"body" : {
"data" : {
"title": "MoCo",
"url" : "http://mozilla.com"
"added_by": "FxOS",
}
}
},
{
"method" : "PATCH",
"path" : "/articles/409",
"body" : {
"data" : {
"read_position" : 3477
}
}
"headers" : {
"Response-Behavior": "light"
}
}
]
}
The response body is a list of all responses:
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Retry-After, Content-Length, Alert, Backoff
Content-Length: 1674
Date: Wed, 17 Feb 2016 18:44:39 GMT
Server: waitress
{
"responses": [
{
"status": 201,
"path" : "/articles",
"body" : {
"data" : {
"id": 411,
"title": "MoFo",
"url" : "http://mozilla.org",
...
}
},
"headers": {
...
}
},
{
"status": 201,
"path" : "/articles",
"body" : {
"data" : {
"id": 412,
"title": "MoCo",
"url" : "http://mozilla.com",
...
}
},
"headers": {
...
}
},
{
"status": 200,
"path" : "/articles/409",
"body" : {
"data" : {
"id": 409,
"url": "...",
...
"read_position" : 3477
}
},
"headers": {
...
}
}
]
}
HTTP Status Codes¶
200 OK
: The request has been processed400 Bad Request
: The request body is invalid50X
: One of the sub-request has failed with a50X
status
Warning
Since the requests bodies are necessarily mappings, posting arbitrary data (like raw text or binary) is not supported.
Note
Responses are executed and provided in the same order than requests.
About transactions¶
The whole batch of requests is executed under one transaction only.
In order words, if one of the sub-request fails with a 503 status for example, then every previous operation is rolled back.
Important
With the current implementation, if a sub-request fails with a 4XX status
(eg. 412 Precondition failed
or 403 Unauthorized
for example) the
transaction is not rolled back.
Server timestamps¶
In order to avoid race conditions, each change is guaranteed to increment the timestamp of the related collection. If two changes happen at the same millisecond, they will still have two different timestamps.
The ETag
header with the current timestamp of the collection for
the current user will be given on collection endpoints.
ETag: "1432208041618"
On record enpoints, the ETag
header value will contain the timestamp of the
record.
In order to bypass costly and error-prone HTTP date parsing, ETag
headers
are not HTTP date values.
A human readable version of the timestamp (rounded to second) is provided though
in the Last-Modified
response headers:
Last-Modified: Wed May 20 17:22:38 2015 +0200
Changed in version 2.0: In previous versions, cache and concurrency control was handled using
If-Modified-Since
and If-Unmodified-Since
. But since the HTTP date
does not include milliseconds, they contained the milliseconds timestamp as
integer. The current version using ETag
is HTTP compliant (see
original discussion.)
Note
The client may send If-Unmodified-Since
or If-Modified-Since
requests
headers, but in the current implementation, they will be ignored.
Important
When collection is empty, its timestamp remains the same until new records are created.
Cache control¶
In order to check that the client version has not changed, a If-None-Match
request header can be used. If the response is 304 Not Modified
then
the cached version is still good.
GET | |
---|---|
If-None-Match: “<timestamp>”
|
|
Changed meanwhile | Return response content |
Not changed | Empty HTTP 304 |
Concurrency control¶
In order to prevent race conditions, like overwriting changes occured in the interim for example,
a If-Match: "timestamp"
request header can be used. If the response is 412 Precondition failed
then the resource has changed meanwhile.
Concurrency control also allows to make sure a creation won’t overwrite any record using
the If-None-Match: *
request header.
The following table gives a summary of the expected behaviour of a resource:
POST | PUT | PATCH | DELETE | |
---|---|---|---|---|
If-Match: “timestamp”
|
||||
Changed meanwhile | HTTP 412 |
HTTP 412 |
HTTP 412 |
HTTP 412 |
Not changed | Create | Overwrite | Modify | Delete |
If-None-Match: *
|
||||
Id exists | HTTP 412 |
HTTP 412 |
No effect | No effect |
Id unknown | Create | Create | No effect | No effect |
When the client receives a 412 Precondition failed
, it can then choose to:
- overwrite by repeating the request without concurrency control;
- reconcile the resource by fetching, merging and repeating the request.
Replication¶
In order to replicate the timestamps when importing existing records, it is possible to force the last modified values.
When a record is created (via POST or PUT), the specified timestamp becomes the new collection timestamp if it is in the future (i.e. greater than current one). If it is in the past, the record is created with the timestamp in the past but the collection timestamp is bumped into the future as usual.
When a record is replaced, modified or deleted, if the specified timestamp is less or equal than the existing record, the value is simply ignored and the timestamp is bumped into the future as usual.
Backoff indicators¶
Backoff header on heavy load¶
A Backoff
header will be added to the success responses (>=200 and
<400) when the server is under heavy load. It provides the client with
a number of seconds during which it should avoid doing unnecessary
requests.
Backoff: 30
Note
The back-off time is configurable on the server.
Note
In other implementations at Mozilla, there was
X-Weave-Backoff
and X-Backoff
but the X-
prefix for
header has been deprecated since.
Retry-After indicators¶
A Retry-After
header will be added if response is an error (>=500).
See more details about error responses.
Error responses¶
Protocol description¶
Every response is JSON.
If the HTTP status is not OK (<200 or >=400), the response contains a JSON mapping, with the following attributes:
code
: matches the HTTP status code (e.g400
)errno
: stable application-level error number (e.g.109
)error
: string description of error type (e.g."Bad request"
)message
: context information (e.g."Invalid request parameters"
)info
: online resource (e.g. URL to error details)details
: additional details (e.g. list of validation errors)
Example response
{
"code": 412,
"errno": 114,
"error": "Precondition Failed",
"message": "Resource was modified meanwhile",
"info": "https://server/docs/api.html#errors",
}
Refer yourself to the ref:set of errors codes <errors>.
Retry-After indicators¶
A Retry-After
header will be added to error responses (>=500),
telling the client how many seconds it should wait before trying
again.
Retry-After: 30
Precondition errors¶
As detailed in the timestamps section, it is
possible to add concurrency control using ETag
request headers.
When a concurrency error occurs, a 412 Precondition Failed
error response
is returned.
Additional information about the record currently stored on the server will be
provided in the details
field:
{
"code": 412,
"errno": 114,
"error":"Precondition Failed"
"message": "Resource was modified meanwhile",
"details": {
"existing": {
"last_modified": 1436434441550,
"id": "00dd028f-16f7-4755-ab0d-e0dc0cb5da92",
"title": "Original title"
}
},
}
Conflict errors¶
When a record violates unicity constraints, a 409 Conflict
error response
is returned.
Additional information about conflicting record and field name will be
provided in the details
field.
{
"code": 409,
"errno": 122,
"error": "Conflict",
"message": "Conflict of field url on record eyjafjallajokull"
"info": "https://server/docs/api.html#errors",
"details": {
"field": "url",
"record": {
"id": "eyjafjallajokull",
"last_modified": 1430140411480,
"url": "http://mozilla.org"
}
}
}
Validation errors¶
When multiple validation errors occur on a request, the first one is presented in the message.
The full list of validation errors is provided in the details
field.
{
"code": 400,
"errno": 109,
"error": "Bad Request",
"message": "Invalid posted data",
"info": "https://server/docs/api.html#errors",
"details": [
{
"description": "42 is not a string: {'name': ''}",
"location": "body",
"name": "name"
}
]
}
Deprecation¶
A track of the client version will be kept to know after which date each old version can be shutdown.
The date of the end of support is provided in the API root URL (e.g. /v0
)
Using the Alert
response header, the server can communicate any potential warning
messages, information, or other alerts.
The value is JSON mapping with the following attributes:
code
: one of the strings"soft-eol"
or"hard-eol"
;message
: a human-readable message (optional);url
: a URL at which more information is available (optional).
A 410 Gone
error response can be returned if the
client version is too old, or the service had been remplaced with
a new and better service using a new protocol version.
See details in Recommended settings to activate deprecation.
Syncto specific headers¶
Syncto also expose Firefox Sync information such as:
Quota-Remaining
: Remaining KB for the user collection if Quota are enabled on the user sync server.
Troubleshooting¶
We are doing the best we can so you do not have to read this section.
That said, we have included solutions (or at least explanations) for some common problems below.
If you do not find a solution to your problem here, please ask for help!
OpenSSL error when installing on Mac OS X¶
#include <openssl/aes.h>
^
1 error generated.
error: command 'clang' failed with exit status 1
Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries, please refer to the installation documentation to fix this.
Contributing¶
Thank you for considering to contribute to Syncto!
note: | No contribution is too small; please submit as many fixes for typos and grammar bloopers as you can! |
---|---|
note: | Open a pull-request even if your contribution is not ready yet! It can be discussed and improved collaboratively! |
Run tests¶
make tests
Cleaning your environment¶
There are three levels of cleaning your environment:
make clean
will remove*.pyc
files and__pycache__
directory.make distclean
will also remove*.egg-info
files and*.egg
,build
anddist
directories.make maintainer-clean
will also remove the.tox
and the.venv
directories.
Communication channels¶
- Our IRC channel
#fxos-sync
onirc.mozilla.org
— Click here to access the web client! - Our team blog http://www.servicedenuages.fr/
CHANGELOG¶
This document describes changes between each past release.
1.6.0 (unreleased)¶
- Nothing changed yet.
1.5.0 (2016-01-27)¶
- Upgraded to Cliquet 2.15.0
Protocol
- Make sure batch always return 200 except for 5xx errors. (#78)
Bug fixes
- Fix
If-None-Match
header format which doesn’t take quote around the*
parameter. (#76)
Internal changes
- Add a Dockerfile (#77)
- Remove documentation warnings (#74)
1.4.0 (2015-11-17)¶
- Upgraded to Cliquet 2.11.0
New Features
- Pass User-Agent header to sync. (#68)
- Add trusted certificate pinning support. (#72)
See also *Cliquet* changes
1.3.0 (2015-10-27)¶
- Upgraded to Cliquet 2.9.0
Protocol
- Client-state id should now be provided through the bucket id in the URL (#62)
1.2.0 (2015-10-22)¶
- Send
Cache-Control: no-cache
header (#54) - Make sure collection_list return an empty list (#56)
1.1.0 (2015-10-14)¶
- Do not install postgresql dependencies by default.
- Add statsd metrics on SyncClient response status_code. (#49)
- Handle the new Firefox Sync sort=oldest parameter. (#46)
- Rename ids to in_ids to reflect the Kinto protocol. (#50)
- Make sure Next-Page header keeps QueryString parameters. (#47)
- Add a Token server heartbeat. (#44)
- Remove the not accurate Total-Records header when paginating. (#43)
- Expose the now deprecated cliquet.batch_max_requests settings. (#48)
1.0.0 (2015-10-06)¶
- First implementation of Syncto server.
- Connection with Token server and Sync servers.
- Encrypted credentials caching (#30, #31)
- Collections are Read-only by default
- Write permission on collection can be configured.
- Statsd monitoring for backends calls.
- Convert Syncto requests headers to Firefox Sync ones.
- Convert Firefox Sync headers to Syncto ones.