Hi Community,
I have some doubt about how can i put my web app of Shiny in my server ODK.
Is any step anyone can share with me in order to run the application from my server?
Thanks,
Agustin.
1 Like
Thanks for asking. Here are some early thoughts, but would be interested in hearing more from IT experts on this topic (I am going for something simpler as lacking implementation resources and not working within a VPN, which means I cannot deploy the Shiny App on the Internet without a strong Authentication layer). It is also worth checking the following threads on the RStudio community forum.
-
RShiny App installed in a Docker container
-
If your RShiny App is installed on the same server that already hosts ODK Central, you need to allocate a different IP / port to your new Docker container.
1 Like
Hi,
Thanks for your help!
I heard about RuODK, do you know anything about this API? I am looking for help to install it.
Hi Agustin,
pasting your screenshot here:
Breaking down the error:
- You tried to install ruODK, I assume you followed https://docs.ropensci.org/ruODK/#install
- ruODK failed to install because leafem failed to install because raster failed to install.
- Try and install raster via CRAN, then via GitHub. Address the installation errors until it works.
Some resources:
https://github.com/ropensci/ruODK/blob/main/tic.R are the installation steps I use to build the automated test environments. As dependency packages mature, their installation bugs come and go, so this list always changes a bit.
To force install ruODK with all dependencies, run
remotes::install_github(
"ropensci/ruODK@main",
dependencies = TRUE,
upgrade = "always",
build_vignettes = FALSE
)
Try the above and feel free to report back here with any further build errors!
I am planning on doing something similar. My take on it is that it would be best (from a security standpoint) to create a new docker container with the shiny server/app with its own address:port and this then communicates with the ODK database over the docker network as a data source. Then add this container to the docker run script for ODK so it starts up with everything else.
Thoughts?
There is no simple way to provide a live feed for Shiny apps as far as I know. You will need to download the ODK data often enough to be current and place it where the shiny app can consume it.
@mathieubossaert and @Thalie have worked in greater detail on this issue. Mathieu built some gruelling SQL operations directly in ODK Central's PostGIS DB to extract the data. I'm lazy and just use the API via ruODK.
My two-step process is
- Have a script that downloads the data and attachments, then parses the data and saves it as an .RData file into some folder. ruODK parses the data such that images are hotlinked after download.
- Have a Shiny app running on that server (or as a Docker image, mounting that data folder) which opens the .RData and uses the file paths in the media fields to hotlink the images and other media.
This keeps all my ODK credentials and data outside of the Shiny app.
If that's not an option and you want the download to happen from inside the Shiny app:
- ODK credentials need to be available to the Shiny Server, but secured from access. See https://gargle.r-lib.org/articles/articles/managing-tokens-securely.html for some inspiration.
- Downloading the data could take longer than the user will want to wait. Do you want to download on access, or download on schedule, or in the background while displaying previously downloaded data from a .RData save file?
- Downloading data is one thing, downloading attachments takes even longer. Consider something like
ruODK::odata_submission_get(download=FALSE)orruODK::submission_export(attachments=FALSE)to get just the data without the attachments. - You might want to persist the locally downloaded data in a docker volume or mount. You could place the save file of the downloaded and parsed ODK data into that mounted directory and even do so from outside the Shiny app. That would also keep ODK credentials away from the Shiny app.
Let us know how you go, would be interesting to learn what worked for you in the end!
1 Like
We first tried and achievied to query Central's DB directly with PostgreSQL FDW (as we always did with Aggregate) and xml capabilities, to create views exposing submitted data "live and direct" but :
So we now use the OData API calls to feed and update our internal database.
As we can now filter the API to get the lasts json submissions (filter over submission_date), the refresh frequency (cron job) can be "high" (15 minutes here to be sure that submitted data will be seen in our other tools), but is not a real "live and direct" access.
So we have quite the same process as @Florian_May :
- first feeding a json database table with new submissions, parse it with SQL to expose tables or views
- and then connecting other apps to show data and media
We really start it in a "production" mode this month, because last field season was supported by Aggregate and also because we merged with our neighbors and the team grow up.
I think the API encourage us to be lazy 
and it is comfortable. In a near future, the API will allow us to get only one json for a whole form (including repeats) simplifying again the process.
1 Like
Already does!
You'll get the submissions with all nested data, which requires knowledge of the data structure. This is hard to support in a generic way for ruODK, but easy for the end user who knows their form.
As shown in the vignette, you'll have to get a list of submission IDs first, but that also allows you to determine which ones are already downloaded.
I was thinking about the OData $expand query parameter of the API.
and the recently closed issue : https://github.com/getodk/central/issues/206
1 Like
I had previously developed a demo minimalist Shiny app that relies on the RuODK package that @Florian_May has developed to retrieve the data in real-time from ODK Central. The data retrieval takes a bit of time on my test database (~10s for 6 forms, the largest database having 20 submissions with 219 columns), but I have not tried to optimise it and I did not develop specific performance tests, so I cannot tell you how the Shiny app will behave with larger databases. Shiny apps are something I would like to investigate more (with Flask, Dash or Django apps in Python), but I unfortunately just do not have enough time to dedicate to this and had to prioritise the generation of markdown reports over shiny dashboards, since this is a much simpler and quicker way to visualise data.
In practice what I had done is to create a R package in which I defined generic core Shiny components (modules). This way you can create several Shiny apps building from these core components and store the server and user interface definition of each app in a specific folder in the inst/shiny/ folder of my package, e.g., inst/shiny/app1, inst/shiny/app2. ruODK is only called in in the server definition of the Shiny app. When a package is installed, everything in the inst/ folder is copied into the top-level package directory. About the core Shiny components themselves, you can have a look at a few examples (..._module.R files). The file run_app.R contains the calls to the different Shiny apps, so that when the package is installed you can simply use timci::run_mini_app() to launch your app.
run_mini_app <- function() {
appDir <- system.file("shiny", "mini", package = "timci")
if (appDir == "") {
stop("Could not find `mini` directory. Try re-installing `timci`.", call. = FALSE)
}
shiny::runApp(appDir,
launch.browser = TRUE,
display.mode = "normal")
}
server.R
# Defines the server logic for the minimalist Shiny web application
#
# Find out more about building applications with Shiny here:
# http://shiny.rstudio.com/
library(ruODK)
library(hash)
server <- function(input, output) {
#######################################
# ODK CENTRAL SERVER CONNECTION SETUP #
#######################################
# Connect to the ODK Central server using ruODK
ruODK::ru_setup(
svc = Sys.getenv("ODKC_SVC"),
un = Sys.getenv("ODKC_UN"),
pw = Sys.getenv("ODKC_PW"),
tz = Sys.getenv("TZ"),
verbose = FALSE # Can be switched to TRUE for demo or debugging
)
################
# RAW ODK DATA #
################
# Load ODK forms that have at least 1 submission
odk_form_list <- ruODK::form_list()
valid_odk_forms <- subset(odk_form_list, submissions > 0, select = c(fid))
valid_odk_form_list <- valid_odk_forms$fid
# Load ODK data
odk_data <- hash::hash()
for (form in valid_odk_form_list) {
odk_data[[form]] <- ruODK::odata_submission_get(fid = form)
}
# Execute the form selection module
current_odk_form <- timci::select_list_item_server("select_odk_form",
valid_odk_form_list,
odk_data)
# Execute the info module about the raw ODK data
timci::reactive_odk_data_info_server("raw_odk_info", current_odk_form)
# Execute the table module that displays the raw ODK data
timci::reactive_data_table_server("raw_odk_table", current_odk_form)
# Execute the CSV and Excel download modules
timci::reactive_csv_download_server("raw_odk_csv_export", current_odk_form)
timci::reactive_xlsx_download_server("raw_odk_xlsx_export", current_odk_form)
}
ui.R
# Defines the UI for the minimalist Shiny web application
#
# Find out more about building applications with Shiny here:
# http://shiny.rstudio.com/
# Define UI
ui <- shiny::fluidPage(
shiny::navbarPage("TIMCI Dashboard", id="nav",
shiny::navbarMenu("Data",
# Raw ODK data panel (will be hidden / access restricted)
shiny::tabPanel("ODK data",
# Sidebar layout with input and output definitions
shiny::sidebarLayout(
# Sidebar panel for inputs
shiny::sidebarPanel(
# Help text
shiny::helpText("ODK form selection"),
# Display the list item selection module
timci::select_list_item_ui("select_odk_form"),
# Info text
shiny::p(shiny::strong("Database information")),
# Study data info
timci::odk_data_info_ui("raw_odk_info"),
# Export study data in *.csv format
timci::csv_download_ui("raw_odk_csv_export"),
# Export study data in *.xlsx format
timci::xlsx_download_ui("raw_odk_xlsx_export")),
# Display the table module
shiny::mainPanel(
shiny::fluidRow(
timci::data_table_ui("raw_odk_table"))))
)
)
)
)
Minimalist Shiny app (obviously the real interest is in adding more modules to this minimalist example)
7 Likes
Wow. Neat!
I am so gonna steal this.
Hi guys, time ago i do not write you. Sorry i had some personal problems and i could not give atention to the project.
Once i want to install the RuODK in Rstudio i receive the following error. Anyone knows what is it happen?
Hi @Agustin,
The third last line says that packages xml2 and httr failed to install. Try and install these separately and address any errors, then install ruODK again.
Hi Floria,
Thanks, i could install it. Now i want to advance with the data collect.
I am following the instruction you did in https://docs.ropensci.org/ruODK/ but i am stuck with the first part.
Once i introduce the code in R i receive the following message in the console. But then, i don't understand how can i import the data directly from the server.
One option is to download a CSV to my desktop and then import it from R but i want to do it by the way you explain.
Thanks for your help.
Agustin.
Hey that's great news!
I believe the OData path might be best for your use case. ruODK provides two helpers:
First, the full worked example is in this vignette: https://docs.ropensci.org/ruODK/articles/odata-api.html
Second, Iif you use Rstudio, you can create a new RMarkdown document from the ruODK template "OData". This has the simple steps in it, just fill in the blanks as directed. The Readme also has the command to start this Rmd template without RStudio.
Thanks Florian. I tried to do it but know i have problems with the server, i cant enter via URL.
Attached you can find the code. Do you think that anything is wrong and cause any problem to the server id ODK central?
Thanks again.
test.Rmd.zip (997 Bytes)
Did you set up your ruODK credentials correctly as per the setup vignette?
No, i did not. What do you recomend to go back and repair the server?
Edit:
Read your attached Rmd.zip and it looks like that's not the ruODK template. You shouldn't need to load the packaged example data if you want to analyse your own.
Start over and create a fresh RMarkdown from the ruODK template:
Option 1: Using RStudio, follow https://bookdown.org/yihui/rmarkdown/document-templates.html to create a new Rmarkdown document and select the ruODK template "ODK Central via OData", or
Option 2: Run this command:
rmarkdown::draft("my_analysis.Rmd", template = "odata", package = "ruODK")
Then follow the instructions inside, delete what you don't need, and report back here if you get stuck.
Notably, you should place your credentials into the .Renviron file (instructions in the template, so I won't repeat those here).
You've done well, you're nearly there!
1 Like
On a practical note, most of the installation issues my collaborators or I have encountered were related to an incorrect setup / .Renviron file, so I systematically check that:
- I have an internet connection
- my
.Renvironfile is recognised as such (no hidden extension) and placed in my current working directory => e.g., when using a batch
- my ODK credentials are correct (double check by logging to the ODK Central web interface)
- my Odata service URL is correct (i.e. a project and a form with the corresponding IDs exist on the ODK Central server)
Once you are there, the rest should be pretty smooth thanks to the very extensive doc that Florian has developed.
2 Likes