Server pools and load balancing

https://causalmap.shinyapps.io/Qualia is actually a simple load balancer which redirects traffic to one server from one of two pools of servers.

There are at the moment three servers in the bot pool and just one in the control pool.

The control servers are intended for users who want to edit or administer interviews.

If the URL asks for a specific interview eg ?interview=school , the load balancer redirects to a random one of the bot servers.

So https://causalmap.shinyapps.io/Qualia/?interview=school redirects (randomly) to any of the pool of bot servers like https://causalmap.shinyapps.io/Qualia-01/ and ?interview=school is added back in, e.g. https://causalmap.shinyapps.io/Qualia01/?interview=school

At the moment, all the servers in the two pools have the same code (they are the same app), but additional parts of the UI (for editing etc) are not created or not exposed if the user has a specific interview in the URL eg ?interview=school .

Otherwise the user is redirected (randomly) to any of the pool of control servers like https://causalmap.shinyapps.io/Qualia-control-01. At the moment there is only one control server.

If someone goes directly to one of the pool servers like https://causalmap.shinyapps.io/Qualia-control-01, with or without asking for a specific interview, this will work. But we recommend always directing people to the main load balancer, https://causalmap.shinyapps.io/Qualia. The links shown in the control panel do this.

There is a development server https://causalmap.shinyapps.io/Qualia-dev/ which we will update more frequently, for testing. It can be used for both editing and testing interviews.

Why do we use only 900 dimensions of the text-embedding-3-large embeddings data instead of the 1536 which OpenAI provides?

This model performs very well when truncating the dimensions. This is good because we can use an InnoDB storage engine, which otherwise doesn’t have enough dimensions, rather than MyISAM which does but is slower.