API problems with data.fields objects missing or not including field name.

Bug Reports
1

Hello

The docs for the endpoint Get Tables in the Softr Database API (Get Tables - Softr Documentation) clearly include a fields array within the JSON response. However, when you call that endpoint you don’t get a fields array at all.

There does not seem to be anyway to reliably call a list of the fields (including ID, name, type etc.) of one or more tables in the API.

The use case is as follows:

  1. We want to warehouse the data in our Softr Database within PGSQL (Supabase) to enable centralised reporting alongside other systems and data.
  2. To do this reliably, we need to use an ETL system to extract the data. We use Airbyte.
  3. When interacting with a REST API source, Airbyte will create tables in the destination with column names that match the object attributes returned in the source API’s JSON body.
  4. When you pull a record via the Get Records endpoint, the fields array only includes the ID and the value. There is no field name.
  5. So, left alone our ETL system would create a destination table of data where the column names are simply Softr field IDs. This would not be usable.

To correct this issue, we either need:

  • IDEAL - field names as well as IDs to be included in the data.fields object within the Get Records / Get Single Record API endpoints. This would allow an ETL system like Airbyte to use the field name as the destination column name, not the field ID.
  • WORKAROUND - a reliable way to call fields for a given table, with data that includes the field name. In this case, we could maintain a separate destination table of all fields in the Softr Database, and then use PGSQL views to map field (column) IDs to Softr field names, creating useable views / reports.

This feels pretty fundamental and currently, if we cannot resolve this it would eliminate the commercial use case for Softr for us.

Can we expand the data.fields object in Get Records / Get Single Record to include field names as a minimum?

Aaron

2

@aferguson you could go table by table and get the fields… this being said we could extend and return fields in the tables endpoint too

3

@aferguson please check we have enabled what you have asked

1 Like
4

Thank you @artur for enabling this - it’s really helpful and working perfectly.

It is possible to filter the get records endpoint by updatedat? The use case is an incremental sync. The API docs don’t show an updatedat parameter.

5

@aferguson afaik yes it’s possible but even if not just add one field for updated_at and filter by that

6

Thanks - we can certainly filter client side but that means for each API call we have to pull all data. As databases grow that’s potentially a lot of records. At high sync cadences (say every 5 minutes) that puts a lot of pressure on the API.

It would be easier to filter server side, i.e. filter the API call using updated_at, so we’re only ever pulling new or changed records to sync.

7

@aferguson I mean adding updated_at field in the DB and filter with that field still server side…