WebSocket, Shrek, and AsyncAPI - An Opinionated Intro

Lukasz Gornicki

ยท14 min read

This is a pretty subjective post. I'm sharing my perspective, taking into account years of experience building backend and frontend with user experience in mind.

If you do not want to read this article, then watch the recording of the live stream about the same:

Everything we hear is an opinion, not a fact. Everything we see is a perspective, not the truth. โ€• Marcus Aurelius

This blog post is the first of a series of blog posts about WebSocket I'm working on.

What is WebSocket

It is a pretty old protocol used for duplex communication over TCP connection. It was standardized in 2011. Yes, ten years ago means it is old, super old.

So why do I even mention it in 2021?

It is very widely adopted and will not go away anytime soon because tooling support is excellent and serves its purpose well. Just remind yourself when HTTP/2 showed up and how many years it took everyone to migrate. It would not happen without the strong support and push from all the big players.

Sure, there is HTTP/2 multiplexing and protocols like Mercure or GraphQL Subscription. There is also RFC8441 for WebSocket and HTTP/2 and some tools already adopted it, like Envoy or Jetty. Nevertheless, WebSocket is here to stay.

Anyway, the future of WebSocket has nothing to do with this post. This post is for the AsyncAPI community looking into the AsyncAPI spec because of WebSockets now, no matter the protocol's future.

Websocket use case

  • Do you like to see in Slack that someone is typing a response?
  • Do you like it when a user interface updates without page refresh?
  • Do you like it when your client app knows there are updates available for display?

That is what WebSocket is for. You establish a long-living connection between client and server. Through such a connection, the client can send a stream of messages to the server, and this is possible the other way around at the same time.

One could say: I don't need WebSocket to achieve that. I could just set up a data polling with REST API. Just ask the API every few seconds if there are updates.

Sadly this is not a joke. Engineers do it. Some engineers just take shortcuts, mostly because deadlines hunt them down.

HTTP polling was presented very well in Shrek's famous Are we there yet? scene.

Don't go that path. Do not perform unnecessary connections to your servers and create more and more traffic with more and more resource consumption. Wasting resources is bad and makes Shrek angry. WebSocket changes a lot there:

Figure 1: HTTP Pull vs WebSocket vs Shrek.

Figure 1: HTTP Pull vs WebSocket vs Shrek.

Why AsyncAPI

When building a WebSocket API on a server, you might have some additional needs:

  • Want to document the API for the team that writes a client app, Web UI, Desktop app, or Mobile app.
  • Want to have a way to specify the format of the messages that the server supports to validate them in the runtime.
  • Want to generate a server or/and a client? If not for final production use, then for sure for prototyping and testing.

These are just a few common needs. For WebSocket, you only establish a connection over HTTP protocol, and the rest goes over WS, so OpenAPI specification won't help you much here. WebSocket is one of the patterns in event-based systems. In the end, it is all about a stream of messages and asynchronous processing. Yes, it would be best to use AsyncAPI ๐Ÿ˜ƒ

WebSocket described with AsyncAPI

When I google for some public WebSocket API to play with, I find mostly currency trading products:

Currency trading is a topic I know nothing about ๐Ÿคทโ€โ™‚ but it feels interesting to explore more. Documentation of the 1st and 2nd API looks familiar from look&feel perspective. I think we can make a bet they are already using AsyncAPI, and Kraken most probably is still running on version 1. Let's release the Kraken then.

I'm sorry if you expected me to describe Shrek's API interface using AsyncAPI. It would be fun, but only fun, and I'd also like to teach you something.

I will write an AsyncAPI document for Kraken API after playing with the API and basing it on the current documentation.

Playing with WebSocket API

The best way to play with a WebSocket API is through a CLI. Who didn't hear about curl in the REST API world? For WebSocket, I would recommend websocat. Kraken's API is partially public without authorization which is just great because to play with it, you do not have to set up an account to get an authorization token.

  1. Install websocat. For other installation options, check out this list.
    brew install websocat
  2. Establish connection with the API:
    websocat wss://ws.kraken.com
  3. Ping the API to see if it responds. Just type the below message and hit Enter:
    {"event": "ping"}
  4. Now subscribe to the event ticker stream that sends messages with currency price. Just type the below message and hit Enter:
    {  "event": "subscribe",  "pair": [    "XBT/USD",    "XBT/EUR"  ],  "subscription": {    "name": "ticker"  }}
  5. You should now see a constant stream of data sent by the server. You do not have to ask the API every second for an update, as the update is pushed to you.

Boy, it is always such fun to do it. Like seriously, I always have fun playing with APIs, any APIs. Just making this API "conversation". I hope nothing is wrong with me ๐Ÿ˜…

Now you know how to interact with the Kraken API. Now let's try to describe it using AsyncAPI.

Describing API using AsyncAPI

I'll explain, in detail, how to describe Websocket API with AsyncAPI in another blog post that will be part of the series. Why? I don't want to make this post super lengthy and discourage others from reading it. Let us learn step by step.

For now, I will throw here a full AsyncAPI document I created for the Kraken API. You can also open it up in the AsyncAPI Studio and compare with their current documentation

Familiarize with below before you look at the AsyncAPI document:

  • AsyncAPI describes the API interface between the client and the server. In other words, the AsyncAPI document is for the user of the API. It does not describe what the server does but what the user can do with the API.
  • Kraken API is quite complex. It has some beta servers, some private messages, and messages closely related to vocabulary specific for currency trading. I dropped all of those from my research not to overcomplicate things. In other words, the AsyncAPI file that you can see below is not a complete document.
  • Websocket protocol is very flexible, and therefore you can implement the server in many different ways. There is no standard way of doing things, like there is no common way of doing things with AsyncAPI. We can only make some generic assumptions looking at existing implementations:
    • Your server has one entry point, just one endpoint that you communicate with to gain access to the API. It can be a path with some dynamic values, as some data id. It can also be nothing, no path at all, like in the case of below Kraken API. These entry points are channels in AsyncAPI document. Commonly, Websocket API has just one channel that user can send messages to and receive messages at the same time
    • AsyncAPI publish and subscribe operations translates to messages user can send to the API and messages user will receive from the API. Depending on API complexity, sometimes you have an API that sends only one message. You can also have a situation where you can send to the server multiple different messages, and also receive different messages in response. This is when you need to use oneOf as I did in document for Kraken API.
  • Current AsyncAPI limitation is that you cannot specify that once the user sends (publish) message ping, the pong message is a reply. Look at this thread to participate in an ongoing discussion about request/reply pattern support in AsyncAPI. In the below document, you will notice that for such a use case, I use AsyncAPI specification extensions (x-response).

Message to Kraken API developers and technical writers
In case you want to continue the work I started on the AsyncAPI document for Kraken API, feel free to do that. I'm happy to help, just let me know. Reach me out in our AsyncAPI Slack workspace.

1asyncapi: 2.0.0
4  title: Kraken Websockets API
5  version: '1.8.0'
6  description: |
7    WebSockets API offers real-time market data updates. WebSockets is a bidirectional protocol offering fastest real-time data, helping you build real-time applications. The public message types presented below do not require authentication. Private-data messages can be subscribed on a separate authenticated endpoint. 
9    ### General Considerations
11    - TLS with SNI (Server Name Indication) is required in order to establish a Kraken WebSockets API connection. See Cloudflare's [What is SNI?](https://www.cloudflare.com/learning/ssl/what-is-sni/) guide for more details.
12    - All messages sent and received via WebSockets are encoded in JSON format
13    - All decimal fields (including timestamps) are quoted to preserve precision.
14    - Timestamps should not be considered unique and not be considered as aliases for transaction IDs. Also, the granularity of timestamps is not representative of transaction rates.
15    - At least one private message should be subscribed to keep the authenticated client connection open.
16    - Please use REST API endpoint [AssetPairs](https://www.kraken.com/features/api#get-tradable-pairs) to fetch the list of pairs which can be subscribed via WebSockets API. For example, field 'wsname' gives the supported pairs name which can be used to subscribe.
17    - Cloudflare imposes a connection/re-connection rate limit (per IP address) of approximately 150 attempts per rolling 10 minutes. If this is exceeded, the IP is banned for 10 minutes.
18    - Recommended reconnection behaviour is to (1) attempt reconnection instantly up to a handful of times if the websocket is dropped randomly during normal operation but (2) after maintenance or extended downtime, attempt to reconnect no more quickly than once every 5 seconds. There is no advantage to reconnecting more rapidly after maintenance during cancel_only mode.
21  public:
22    url: ws.kraken.com
23    protocol: wss
24    description: |
25      Public server available without authorization.
26      Once the socket is open you can subscribe to a public channel by sending a subscribe request message.
27  private:
28    url: ws-auth.kraken.com
29    protocol: wss
30    description: |
31      Private server that requires authorization.
32      Once the socket is open you can subscribe to private-data channels by sending an authenticated subscribe request message.
34      The API client must request an authentication "token" via the following REST API endpoint "GetWebSocketsToken" to connect to WebSockets Private endpoints. For more details read https://support.kraken.com/hc/en-us/articles/360034437672-How-to-retrieve-a-WebSocket-authentication-token-Example-code-in-Python-3
36      The resulting token must be provided in the "token" field of any new private WebSocket feed subscription: 
37      ```
38      {
39        "event": "subscribe",
40        "subscription":
41        {
42          "name": "ownTrades",
43          "token": "WW91ciBhdXRoZW50aWNhdGlvbiB0b2tlbiBnb2VzIGhlcmUu"
44        }
45      }
46      ```
49  /:
50    publish:
51      description: Send messages to the API
52      operationId: processReceivedMessage
53      message:
54        oneOf:
55          - $ref: '#/components/messages/ping'
56          - $ref: '#/components/messages/subscribe'
57          - $ref: '#/components/messages/unsubscribe'
59    subscribe:
60      description: Messages that you receive from the API
61      operationId: sendMessage
62      message:
63        oneOf:
64          - $ref: '#/components/messages/pong'
65          - $ref: '#/components/messages/heartbeat'
66          - $ref: '#/components/messages/systemStatus'
67          - $ref: '#/components/messages/subscriptionStatus'
70  messages:
71    ping:
72      summary: Ping server to determine whether connection is alive
73      description: Client can ping server to determine whether connection is alive, server responds with pong. This is an application level ping as opposed to default ping in websockets standard which is server initiated
74      payload:
75        $ref: '#/components/schemas/ping'
76      x-response:
77        $ref: '#/components/messages/pong'
78    heartbeat:
79      description: Server heartbeat sent if no subscription traffic within 1 second (approximately)
80      payload:
81        $ref: '#/components/schemas/heartbeat'
82    pong:
83      summary: Pong is a response to ping message
84      description: Server pong response to a ping to determine whether connection is alive. This is an application level pong as opposed to default pong in websockets standard which is sent by client in response to a ping
85      payload:
86        $ref: '#/components/schemas/pong'
87    systemStatus:
88      description: Status sent on connection or system status changes.
89      payload:
90        $ref: '#/components/schemas/systemStatus'
91      examples:
92        - payload:
93            connectionID: 8628615390848610000
94            event: systemStatus
95            status: online
96            version: 1.0.0
97    subscribe:
98      description: Subscribe to a topic on a single or multiple currency pairs.
99      payload:
100        $ref: '#/components/schemas/subscribe'
101      examples:
102        - payload:
103            event: subscribe
104            pair:
105              - XBT/USD
106              - XBT/EUR
107            subscription:
108              name: ticker
109        - payload:
110            event: subscribe
111            subscription:
112              name: ownTrades
113              token: WW91ciBhdXRoZW50aWNhdGlvbiB0b2tlbiBnb2VzIGhlcmUu
114      x-response:
115        $ref: '#/components/messages/subscriptionStatus'
116    unsubscribe:
117      description: Unsubscribe, can specify a channelID or multiple currency pairs.
118      payload:
119        $ref: '#/components/schemas/subscribe'
120      examples:
121        - payload:
122            event: unsubscribe
123            pair:
124              - XBT/EUR
125              - XBT/USD
126            subscription:
127              name: ticker
128        - payload:
129            event: unsubscribe
130            subscription:
131              name: ownTrades
132              token: WW91ciBhdXRoZW50aWNhdGlvbiB0b2tlbiBnb2VzIGhlcmUu
133      x-response:
134        $ref: '#/components/messages/subscriptionStatus'
135    subscriptionStatus:
136      description: Subscription status response to subscribe, unsubscribe or exchange initiated unsubscribe.
137      payload:
138        $ref: '#/components/schemas/subscriptionStatus'
139      examples:
140        - payload:
141            channelID: 10001
142            channelName: ohlc-5
143            event: subscriptionStatus
144            pair: XBT/EUR
145            reqid: 42
146            status: unsubscribed
147            subscription:
148              interval: 5
149              name: ohlc
150        - payload:
151            errorMessage: Subscription depth not supported
152            event: subscriptionStatus
153            pair: XBT/USD
154            status: error
155            subscription:
156              depth: 42
157              name: book
159  schemas:
160    ping:
161      type: object
162      properties:
163        event:
164          type: string
165          const: ping
166        reqid:
167          $ref: '#/components/schemas/reqid'
168      required:
169        - event
170    heartbeat:
171      type: object
172      properties:
173        event:
174          type: string
175          const: heartbeat
176    pong:
177      type: object
178      properties:
179        event:
180          type: string
181          const: pong
182        reqid:
183          $ref: '#/components/schemas/reqid'
184    systemStatus:
185      type: object
186      properties:
187        event:
188          type: string
189          const: systemStatus
190        connectionID:
191          type: integer
192          description: The ID of the connection
193        status:
194          $ref: '#/components/schemas/status'
195        version:
196          type: string
197    status:
198      type: string
199      enum:
200        - online
201        - maintenance
202        - cancel_only
203        - limit_only
204        - post_only
205    subscribe:
206      type: object
207      properties:
208        event:
209          type: string
210          const: subscribe
211        reqid:
212          $ref: '#/components/schemas/reqid'
213        pair:
214          $ref: '#/components/schemas/pair'
215        subscription:
216          type: object
217          properties:
218            depth:
219              $ref: '#/components/schemas/depth'
220            interval:
221              $ref: '#/components/schemas/interval'
222            name:
223              $ref: '#/components/schemas/name'
224            ratecounter:
225              $ref: '#/components/schemas/ratecounter'
226            snapshot:
227              $ref: '#/components/schemas/snapshot'
228            token:
229              $ref: '#/components/schemas/token'
230          required:
231            - name
232      required:
233        - event
234    unsubscribe:
235      type: object
236      properties:
237        event:
238          type: string
239          const: unsubscribe
240        reqid:
241          $ref: '#/components/schemas/reqid'
242        pair:
243          $ref: '#/components/schemas/pair'
244        subscription:
245          type: object
246          properties:
247            depth:
248              $ref: '#/components/schemas/depth'
249            interval:
250              $ref: '#/components/schemas/interval'
251            name:
252              $ref: '#/components/schemas/name'
253            token:
254              $ref: '#/components/schemas/token'
255          required:
256            - name
257      required:
258        - event
259    subscriptionStatus:
260      type: object
261      oneOf:
262        - $ref: '#/components/schemas/subscriptionStatusError'
263        - $ref: '#/components/schemas/subscriptionStatusSuccess'
264    subscriptionStatusError:
265      allOf:
266        - properties:
267            errorMessage:
268              type: string
269          required:
270            - errorMessage
271        - $ref: '#/components/schemas/subscriptionStatusCommon'
272    subscriptionStatusSuccess:
273      allOf:
274        - properties:
275            channelID:
276              type: integer
277              description: ChannelID on successful subscription, applicable to public messages only.
278            channelName:
279              type: string
280              description: Channel Name on successful subscription. For payloads 'ohlc' and 'book', respective interval or depth will be added as suffix.
281          required:
282            - channelID
283            - channelName
284        - $ref: '#/components/schemas/subscriptionStatusCommon'
285    subscriptionStatusCommon:
286      type: object
287      required:
288         - event
289      properties:
290        event:
291          type: string
292          const: subscriptionStatus
293        reqid:
294          $ref: '#/components/schemas/reqid'
295        pair:
296          $ref: '#/components/schemas/pair'
297        status:
298          $ref: '#/components/schemas/status'
299        subscription:
300          required:
301            - name
302          type: object
303          properties:
304            depth:
305              $ref: '#/components/schemas/depth'
306            interval:
307              $ref: '#/components/schemas/interval'
308            maxratecount:
309              $ref: '#/components/schemas/maxratecount'
310            name:
311              $ref: '#/components/schemas/name'
312            token:
313              $ref: '#/components/schemas/token'
314    interval:
315      type: integer
316      description: Time interval associated with ohlc subscription in minutes.
317      default: 1
318      enum:
319        - 1
320        - 5
321        - 15
322        - 30
323        - 60
324        - 240
325        - 1440
326        - 10080
327        - 21600
328    name:
329      type: string
330      description: The name of the channel you subscribe too.
331      enum:
332        - book
333        - ohlc
334        - openOrders
335        - ownTrades
336        - spread
337        - ticker
338        - trade
339    token:
340      type: string
341      description: base64-encoded authentication token for private-data endpoints.
342    depth:
343      type: integer
344      default: 10
345      enum:
346        - 10
347        - 25
348        - 100
349        - 500
350        - 1000
351      description: Depth associated with book subscription in number of levels each side.
352    maxratecount:
353      type: integer
354      description: Max rate-limit budget. Compare to the ratecounter field in the openOrders updates to check whether you are approaching the rate limit.
355    ratecounter:
356      type: boolean
357      default: false
358      description: Whether to send rate-limit counter in updates (supported only for openOrders subscriptions)
359    snapshot:
360      type: boolean
361      default: true
362      description: Whether to send historical feed data snapshot upon subscription (supported only for ownTrades subscriptions)
363    reqid:
364      type: integer
365      description: client originated ID reflected in response message.
366    pair:
367      type: array
368      description: Array of currency pairs.
369      items:
370        type: string
371        description: Format of each pair is "A/B", where A and B are ISO 4217-A3 for standardized assets and popular unique symbol if not standardized.
372        pattern: '[A-Z\s]+\/[A-Z\s]+'

Personal note
If you can, if you are in a planning phase, new project, etc., then start designing your architecture with AsyncAPI. Don't do the mistake of coding first and then trying to figure out how to describe it with AsyncAPI ๐Ÿ˜…

Stay tuned for the next blog post that guides you step by step through the above document โ˜ฎ๏ธ

I recommend you also read another article from the series about WebSocket: Creating AsyncAPI for WebSocket API - Step by Step.