Importing Questions & Answers

[SHogenboom]

We are exploring the possibility of a transfer from our own Q&A system to Scoold. I cannot currently find any information on whether it is possible to import Q&A's in a automated and standardized format. With over 2,000 questions in our database we are not looking to add each Q&A manually.

Does anybody know where to begin looking for a solution? Has anyone already accomplished this?
Thanks in advance!

[albogdano]

Hi Sally! It is possible to automate that task using the Scoold API. Some glue code is probably required. The API is Swagger compatible so you could generate an API client for almost any programming language. Unfortunately, I don't have a step-by-step guide ready but I've seen other people do the same operation and usually, I would help them along the way.

[SHogenboom]

@albogdano thanks so much for this quick reply! Unfortunately, I have no knowledge whatsoever about using APIs and our IT team is too busy (who isn’t) to look into it.

I did think about another option which I wanted to pitch: create .json files which mimic the Stackoverflow Teams export. Again, I’m not finding any documentation from Scoold other than ‘import the .zip’. Would it work is I created a zip with only posts.json and upload that? It does mention somewhere that a para import will overwrite all existing data - would the same happen if I did a (fake) import from Stackoverflow ?

[albogdano]

If you manage to replicate the JSON schema of the StackOverflow data dump, then Scoold will be able to restore it if you package it as a .zip file. There is an explanation of the schema here: https://meta.stackexchange.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede

The existing data in the Scoold database table will be overwritten if it contains the same ids from the imported backup.
All other objects will remain in the database unchanged.

[SHogenboom]

@albogdano we are still trying out the import in between other jobs. We’ve manage to import the StackOverflow dump, not entirely consistently but it works.

We are struggling however with the fact that the entire import is dumped into a single ‘default’ space. We tried adding a json field with information on the targeted ‘space’, but that did not work. Do you perhaps have any ideas how we could use the StackOverflow import workflow, while assigning questions/answers to their dedicated spaces? Or is this perhaps not a supported feature and would we need to use the API for that as well?

[albogdano]

Here's an example:

1. The space property of each post object should be in the following format "space": "scooldspace:space-id-dashed:Space Name with Spaces"
2. After the import, go to the Administration page in Scoold and actually create the space "Space Name with Spaces"
3. The space should appear in the top right dropdown menu and the imported post should appear in that space when you select it.

Another way would be to use the create spaces API to programmatically create each space you want to attach to posts.~
Hope that helps...

[SHogenboom]

I’m trying to use the API to create/upload the questions. However, something is going wrong. I copied the json schema from a test question which I created via the Scoold UI as hosted on the localserver. I can create a new post; but the question never shows up in the local Scoold overview. When I access the para database via the app overview it does show that new posts were created. Any ideas on what is going wrong?

The json:

{
  "id": "1588125460816990222",
  "timestamp": 1688723200495,
  "type": "posts",
  "appid": " ou-q2q-migration",
  "parentid": null,
  "creatorid": "1587770774985379840:profile",
  "updated": 1688723202308,
  "name": "Test via Scoold API Console",
  "tags": [
    "test"
  ],
  "votes": 0,
  "version": 0,
  "stored": true,
  "indexed": true,
  "cached": true,
  "objectURI": "/posts/1588125460816990222",
  "plural": "postses",
  "answercount": 0,
  "deprecated": false,
  "postLinkForRedirect": "/question/1588125460816990222/test-via-scoold-api-console",
  "viewcount": 0,
  "closed": false,
  "body": "# some text\r\nmore text API",
  "title": "Test via Scoold API Console",
  "space": "scooldspace:default",
  "commentIds": [],
  "key": "ou-q2q-migration_1588125460816990222",
  "tagsString": "test"
}

[albogdano]

This is definitely a problem with the search index. If your Para server is running locally next to Scoold, make sure that the search index is persisted to disk. If you use ParaIO.com then tey rebuildimg the search index.

[SHogenboom]

@albogdano ; I’m sorry I don’t understand what you are referring to. Perhaps a completely different question will help me out. What is the full endpoint URL I need to use to access the API that connects to my para database? I tried a GET request with the following information (I use R):

library("httr2")
endpoint_url <- "https://paraio.com/v1/"

req <- 
  request(endpoint_url) |>
  req_headers("accessKey" = "app:ou-q2a-migration",
              "secretKey” = "MY_SECRET") |>
  req_url_path_append("users") |>    #  generates the url https://paraio.com/v1/users
  req_perform() # equals to a GET request

This returns an authentication problem (HTTP 401 Unauthorized). Which is similar to what I experienced when using the Python client. Basicly, I’m totally confused how to actually make API calls to my para backend from my local R / Python environment.

[albogdano]

There are two authenthication mechanisms used for accessing the API - AWS v4 signed requests and JWT bearer tokens. The Python client for Para uses the former method but also supports the latter with the method paraClient.setAccessToken(jwt).

You can set the endpoint like this:

paraclient = ParaClient('ACCESS_KEY', 'SECRET_KEY');
paraclient.setEndpoint("https://paraio.com")

For details about generating JWTs check the Para docs and search for "super tokens".

[albogdano]

The Para client has all kinds of CRUD methods as well as full text search methods. Don't set headers manually because the client does that for you.