Investigating what it would take to add fiat currency information to BitcoinKnots, I noticed Electrum currently has many implementations, one for eachexchange rate provider, due to lack of a common format for such data.

Therefore, I put together an initial draft of a BIP that could standardisethis so wallets (or other software) and exchange rate providers can simplyinteroperate without a lot of overhead reimplementing the same thing manyways.

One thing I am unsure about, is that currently this draft requires using XBT(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but thosedon't really have a pseudo-ISO currency code to fit in nicely...

Current draft here:https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki

Thoughts? Anything critical missing? Ways to make the interface better?

I am surprised nothing like this exists already, so am all for the idea.

Maybe I am misunderstanding, but I'm not sure you want to havethousand separators and other locale stuff in there. All currenciesincluding USD are often shown in Swedish using space or dot asthousand separator and comma as decimal separator, e.g. "1.234,56 USD"or "1 234,56 USD". I.e. this is something that the locale of theuser's environment defines, not something the server should haveopinions about. It is also not ideal to propose a given format basedon the user's locale, as some users have preferences for this (Ipersonally use US locale for numbers, and a US person who is visitingSweden wouldn't want this to suddenly change).

-Kalle.

On Sat, Mar 4, 2017 at 12:27 AM, Luke Dashjr via bitcoin-dev

Post by Luke Dashjr via bitcoin-devInvestigating what it would take to add fiat currency information to BitcoinKnots, I noticed Electrum currently has many implementations, one for eachexchange rate provider, due to lack of a common format for such data.Therefore, I put together an initial draft of a BIP that could standardisethis so wallets (or other software) and exchange rate providers can simplyinteroperate without a lot of overhead reimplementing the same thing manyways.One thing I am unsure about, is that currently this draft requires using XBT(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but thosedon't really have a pseudo-ISO currency code to fit in nicely...https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawikiThoughts? Anything critical missing? Ways to make the interface better?Luke_______________________________________________bitcoin-dev mailing listhttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

I'm not sure if a BIP is the right thing in that case, given that theprovided functionality is not special to Bitcoin and can be used inother contexts as well.

But ignoring this, the server should be authenticated at aminimum. Otherwise manipulating exchange rates seems to be a niceway for the attacker on the wire to make money...

Apart from that, my feeling is that it could be simplified. Is longpolling useful? And is the historical rate thing really necessaryfor typical applications?

If yes, the client should be allowed to decide on which time scale thedata should be. (tick, min, hour, day, ...) That goes together withclearly defining the type field (something like low, high, open, close,but without flexibility). Think of a candle-stick chart basically.

Also, pushing may be more appropriate for "current" rates than polling.Then no polling interval is necessary. On the other hand, this addscomplexity in other places, e.g., state.

Tim

Post by Luke Dashjr via bitcoin-devInvestigating what it would take to add fiat currency information toBitcoin Knots, I noticed Electrum currently has many implementations, one foreach exchange rate provider, due to lack of a common format for such data.Therefore, I put together an initial draft of a BIP that couldstandardise this so wallets (or other software) and exchange rate providers cansimply interoperate without a lot of overhead reimplementing the same thingmany ways.One thing I am unsure about, is that currently this draft requiresusing XBT (as BTC) for Bitcoin amounts. It would seem nicer to use satoshis,but those don't really have a pseudo-ISO currency code to fit in nicely... https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawikiThoughts? Anything critical missing? Ways to make the interface better?Luke_______________________________________________bitcoin-dev mailing listhttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Post by Tim Ruffing via bitcoin-devBut ignoring this, the server should be authenticated at aminimum. Otherwise manipulating exchange rates seems to be a niceway for the attacker on the wire to make money...

HTTPS would be used for that. It's not something that needs to be at a higherlayer.

When displaying historical transactions, it doesn't really make sense to usethe current market rate, but rather the market rate at the time the paymentwas made. While wallets might simply cache it with the transaction, it wouldbe perhaps nicer if it could be automatically restored for seed-onlyrecoveries. In any case, if a service/wallet doesn't want to provide/usehistorical information, it can simply not implement that part.

Post by Tim Ruffing via bitcoin-devIf yes, the client should be allowed to decide on which time scale thedata should be. (tick, min, hour, day, ...) That goes together withclearly defining the type field (something like low, high, open, close,but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

Post by Tim Ruffing via bitcoin-devAlso, pushing may be more appropriate for "current" rates than polling.Then no polling interval is necessary. On the other hand, this addscomplexity in other places, e.g., state.

I like the BIP. It can reduce workload during implementation on both sides of the API and it allows to show the user more data without implementing tons of proprietary APIs.Itâs not directly Bitcoin specific (example: BIP32 is also not Bitcoin specific), but I think a BIP is the right way for this.

I would think so, at least for Bitcoin since rates can change significantly ina short period of time (or can they anymore? I haven't really watched lately.)

Long polling is a simple push concept that works on most type of network configurations (NAT, proxy, etc.).The only concern I see here is that an public API will quickly fill up the maximum allowed httpd connections.But itâs solvable.

When displaying historical transactions, it doesn't really make sense to usethe current market rate, but rather the market rate at the time the paymentwas made. While wallets might simply cache it with the transaction, it wouldbe perhaps nicer if it could be automatically restored for seed-onlyrecoveries. In any case, if a service/wallet doesn't want to provide/usehistorical information, it can simply not implement that part.

Iâm also not sure how useful historical datapoint are. I donât think the use case where someone wants to restore from a seed and get all exchange rates during the time of the payment is something users are looking for.However, Itâs optional.

Post by Tim Ruffing via bitcoin-devIf yes, the client should be allowed to decide on which time scale thedata should be. (tick, min, hour, day, ...) That goes together withclearly defining the type field (something like low, high, open, close,but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

Post by Tim Ruffing via bitcoin-devAlso, pushing may be more appropriate for "current" rates than polling.Then no polling interval is necessary. On the other hand, this addscomplexity in other places, e.g., state.

Post by Tim Ruffing via bitcoin-devBut ignoring this, the server should be authenticated at aminimum. Otherwise manipulating exchange rates seems to be a niceway for the attacker on the wire to make money...

HTTPS would be used for that. It's not something that needs to be ata higher layer.

Sure, HTTPS is the way to go. But I think that should be required or atleast noted in the BIP, because people could miss easily, e.g., "Idon't need TLS, all the data is public anyway."

Post by Luke Dashjr via bitcoin-devWhen displaying historical transactions, it doesn't really make senseto use the current market rate, but rather the market rate at the time thepayment was made. While wallets might simply cache it with the transaction,it would be perhaps nicer if it could be automatically restored for seed-only recoveries. In any case, if a service/wallet doesn't want toprovide/use historical information, it can simply not implement that part.

Having the rate at the time of payment is indeed very useful, yes.However that requires just a single value per payment, and there is noquery that tells the server "give me the value closest to timestamp t"or similar.Of course the client can download and keep a large part of history andextract the information on its own but I can imagine that not everyclients wants to do that, and also the client does not know in advancethe bounds (from, to) that it must query.

In the current draft the client or the server cannot specifygranularity. If the clients only wants one value per day but for anentire year, then it has to perform many requests or download andprocess a very large response.Also, I think it's okay that the type field allows for arbitrary user-defined values, but it should also have some precisely defined values(e.g. the mentioned low/high/open/close/typical).For example, it's not clear currently what "low" means for a timestamp(as opposed to a time span). Is it the low of the entire day or the lowsince the previous record or something different?

One has to be careful not to add too much complexity though. As soon asone moves away from timestamps to something like hours or days, allkind of issues with timezone, daylight saving time etc. appear. Maybe asimple way to let the client ask "give me one value for every intervalof 3600 seconds" or similar.

Forgot one thing:For longpolling, maybe we would like to have the possibility to requestsome periodic message from the server. Otherwise clients cannotdistinguish between the situations 1. "value is still in the requestedbounds (minrate, maxrate)" and 2. "connection has dropped". So the usermay take a wrong decision because he assumed that the value is stillin bounds holds but actually the server has died.

Post by Tim Ruffing via bitcoin-devFor longpolling, maybe we would like to have the possibility to requestsome periodic message from the server. Otherwise clients cannotdistinguish between the situations 1. "value is still in the requestedbounds (minrate, maxrate)" and 2. "connection has dropped". So the usermay take a wrong decision because he assumed that the value is stillin bounds holds but actually the server has died.

That's the job of TCP.Do you think we need to explicitly specify a keepalive configuration?Seems like that would vary based on client or network.

Post by Tim Ruffing via bitcoin-devHaving the rate at the time of payment is indeed very useful, yes.However that requires just a single value per payment, and there is noquery that tells the server "give me the value closest to timestamp t"or similar.Of course the client can download and keep a large part of history andextract the information on its own but I can imagine that not everyclients wants to do that, and also the client does not know in advancethe bounds (from, to) that it must query.

It would be a privacy leak to request only the specific timestamps, but Isuppose many wallets lack even basic privacy to begin with.

To address the bounds issue, I have specified that when from/to don't have anexact record, that the previous/next (respectively) is provided.

Hopefully this addresses both concerns?

Post by Tim Ruffing via bitcoin-devIn the current draft the client or the server cannot specifygranularity. If the clients only wants one value per day but for anentire year, then it has to perform many requests or download andprocess a very large response.

That's what the "timedelta" field solves, no?If you want one value per day, you'd put 86400.

Post by Tim Ruffing via bitcoin-devAlso, I think it's okay that the type field allows for arbitrary user-defined values, but it should also have some precisely defined values(e.g. the mentioned low/high/open/close/typical).For example, it's not clear currently what "low" means for a timestamp(as opposed to a time span). Is it the low of the entire day or the lowsince the previous record or something different?

Is it not sufficient for the server to specify this in the description of thegiven currency-pair feed?

Post by Luke Dashjr via bitcoin-devIt would be a privacy leak to request only the specific timestamps,but I suppose many wallets lack even basic privacy to begin with.To address the bounds issue, I have specified that when from/to don'thave an exact record, that the previous/next (respectively) is provided.Hopefully this addresses both concerns?

Post by Tim Ruffing via bitcoin-devIn the current draft the client or the server cannot specifygranularity. If the clients only wants one value per day but for anentire year, then it has to perform many requests or download andprocess a very large response.

That's what the "timedelta" field solves, no?If you want one value per day, you'd put 86400.

Is it not sufficient for the server to specify this in thedescription of the given currency-pair feed?

That works but a standardized way of indicating that piece ofinformation to the client is useful. Then the client can display a"connection status" to the user, e.g., an "possible out-of-date"warning like the warning sign in the Qt GUI when Bitcoin Core iscatching up the network.

Post by Tim Ruffing via bitcoin-devThat works but a standardized way of indicating that piece ofinformation to the client is useful. Then the client can display a"connection status" to the user, e.g., an "possible out-of-date"warning like the warning sign in the Qt GUI when Bitcoin Core iscatching up the network.

Wait, forget this reply, I mixed up the two issues of keepalive anddefinition of low, high etc... -.-

1. Keepalive for longpolling:As I said, this can be useful for an out-of-date warning. I don't knowif this is better solved with TCP keepalive or on the higher layer.

2. Definition of low, high:My feeling is that there is nothing wrong with providing exactdefinitions in the BIP, i.e.., giving up the flexibility does not toohurt much. However all of this is a minor issue after all.

Why are multiple results separated by a line-feed rather than using a JSON Array?* Clients ought to cache historical data, and using a line-feed format allows them to simply append to a cache file.* Parsing JSON typically requires the entire data parsed together as a single memory object. Using simple lines to separate results, however, allows parsing a single result at a time.What if long descriptions require line and paragraph breaks?* Clients should word-wrap long lines, and JSON escapes newlines as "\n" which can be used doubly ("\n\n") for paragraph breaks.

I'd file this under premature optimization at the cost ofinteroperability. If you use JSON, then please use it properly.

I'd also say it's the job of the parser to offer a way of doing that,e.g. in .NET you can easily achieve that with Newtonsoft's JSONparser: http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time.

On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev

Investigating what it would take to add fiat currency information to BitcoinKnots, I noticed Electrum currently has many implementations, one for eachexchange rate provider, due to lack of a common format for such data.Therefore, I put together an initial draft of a BIP that could standardisethis so wallets (or other software) and exchange rate providers can simplyinteroperate without a lot of overhead reimplementing the same thing manyways.One thing I am unsure about, is that currently this draft requires using XBT(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but thosedon't really have a pseudo-ISO currency code to fit in nicely...https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawikiThoughts? Anything critical missing? Ways to make the interface better?Luke_______________________________________________bitcoin-dev mailing listhttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

This formatting of JSON isn't unheard of though, it's typically called JSONStreaming[1]. As long as exchanges implementing the API actually follow theBIP and keep one JSON object per line, it shouldn't be a problem to decode.

1. https://en.wikipedia.org/wiki/JSON_Streaming

On Tue, Mar 7, 2017 at 8:07 AM Marcel Jamin via bitcoin-dev <

Why are multiple results separated by a line-feed rather than using a

JSON Array?

* Clients ought to cache historical data, and using a line-feed format

allows them to simply append to a cache file.

* Parsing JSON typically requires the entire data parsed together as a

single memory object. Using simple lines to separate results, however,allows parsing a single result at a time.

What if long descriptions require line and paragraph breaks?* Clients should word-wrap long lines, and JSON escapes newlines as "\n"

which can be used doubly ("\n\n") for paragraph breaks.I'd file this under premature optimization at the cost ofinteroperability. If you use JSON, then please use it properly.I'd also say it's the job of the parser to offer a way of doing that,e.g. in .NET you can easily achieve that with Newtonsoft's JSONhttp://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time.On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev

Investigating what it would take to add fiat currency information to

Bitcoin

Knots, I noticed Electrum currently has many implementations, one for

each

exchange rate provider, due to lack of a common format for such data.Therefore, I put together an initial draft of a BIP that could

standardise

this so wallets (or other software) and exchange rate providers can

simply

interoperate without a lot of overhead reimplementing the same thing manyways.One thing I am unsure about, is that currently this draft requires using

XBT

(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but