Up and running with a Java based greenfield side project

Greenfield-Webapplikation mit Java im Ein-Person-Selbstversuch Alexander Reelsen alr@spinscale.de | @spinscale

Today’s goal How to run and maintain a modern web java application with minimal resources

About me By day: Community @ Elastic By night: sleep In between: Elasticsearch enthusiast, web framework evaluator, serverless observer, distributed work proponent, basketball fan Java since ‘07

Why greenfield? Assessing current skills & technologies Acquiring new skills & technologies Communication/Feedback Business idea Test out competition Understand the full application lifecycle Just for fun

Limitations Full-time job Family & Kids Covid-19 No full stack developer

Requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Requirements Do the needful - but not more Operational simplicity - No Kubernetes! Easy rollouts - git pushes No login handling - OAuth Optimize for (single) developer productivity Small deployments, few dependencies! Ability to debug

Idea Developer Advocate Conference & Talks Management

Conferences @Elastic Labels for tagging/reporting Milestone for CfP closure No fixed date fields No final closing report (pictures, sponsors, competitors) No easy overview over submitted talks

Stats (+Airtable) Participants Youtube links

Topics Database & Persistence Authentication & Security Web framework Frontend Development Environment & Testing Production Environment & Deployment

Topics Database & Persistence (JDBI, Postgres, HikariCP) Authentication & Security (Pac4j) Web framework (Javalin) Frontend (htmx, hyperscript, Apache echarts, tailwindcss) Development Environment & Testing (direnv, testcontainers) Production Environment & Deployment

Database & Persistence Less magic, more SQL

Persistence Layer 1. Elasticsearch using templating to prevent Object Mapping and not take the whole of Elasticsearch as dependency into 2. Postgres/H2 using ebean 3. Postgres/H2 using JDBI

Persistence Layer: Connection Pool PGSimpleDataSource ds = new PGSimpleDataSource(); ds.setUser(System.getenv(“DB_USERNAME”)); ds.setPassword(System.getenv(“DB_PASSWORD”)); ds.setUrl(System.getenv(“DB_URL”)); HikariConfig hc = new HikariConfig(); hc.setDataSource(ds); hc.setMinimumIdle(1); hc.setMaximumPoolSize(6); this.dataSource = new HikariDataSource(hc); this.jdbi = Jdbi.create(dataSource); this.jdbi.installPlugin(new PostgresPlugin()); this.jdbi.installPlugin(new SqlObjectPlugin());

JDBI Jdbi provides convenient, idiomatic access to relational data in Java.

JDBI Jdbi provides convenient, idiomatic access to relational data in Java. Unlike an ORM, we do not aim to provide a complete object relational mapping framework - instead of that hidden complexity, we provide building blocks that allow you to construct the mapping between relations and objects as appropriate for your application.

User entity public final class User { private Long id; … private List<Organization> organizations = new ArrayList<>(1); }

JDBI - Dao interfaces @RegisterBeanMapper(value = User.class, prefix = “u”) @RegisterBeanMapper(value = Organization.class, prefix = “o”) @UseRowReducer(UserOrganizationDocReducer.class) @SqlQuery(“”” SELECT u.id u_id, u.username u_username, u.display_name u_display_name, u.picture_url u_picture_url, u.email u_email, u.created_at u_created_at, u.modified_at u_modified_at, o.id o_id, o.name o_name, o.created_at o_created_at, o.modified_at o_modified_at FROM users u JOIN user_organizations uo ON uo.user_id = u.id JOIN organization o ON uo.organization_id = o.id WHERE u.username = ? “”“) User findByUsername(String username);

JDBI - RowReducer class UserOrganizationDocReducer implements LinkedHashMapRowReducer<Long, User> { @Override public void accumulate(Map<Long, User> map, RowView rv) { User u = map.computeIfAbsent(rv.getColumn(“u_id”, Long.class), id -> rv.getRow(User.class)); } } if (rv.getColumn(“o_id”, Long.class) != null) { u.getOrganizations().add(rv.getRow(Organization.class)); }

JDBI - Custom SQL @SqlQuery(“”” SELECT to_char(DATE_TRUNC(‘month’,created_at), ‘YYYY-MM’) AS month, SUM(count) AS count FROM analytics WHERE organization_id = :organization.id AND created_at >= :from AND created_at <= :to AND type = :type GROUP BY month; “”“) @KeyColumn(“month”) @ValueColumn(“count”) Map<String, Long> sumAndGroupByMonth( @BindBean(“organization”) Organization organization, @Bind(“type”) Analytics.AnalyticsType type, @Bind(“from”) OffsetDateTime from, @Bind(“to”) OffsetDateTime to);

JDBI - enums public final class Comment { @EnumByName public enum CommentType { COMMENT, REPORT }

JDBI - security Multi-tenant system, implicit security issue How to prevent unauthorized access? Every DAO read operation requires an organization argument Organization cannot be injected from outside

Flyway Database Migrations // run the flyway migration before the app starts Flyway flyway = Flyway.configure().dataSource(dataSource).load(); final MigrateResult result = flyway.migrate(); logger.info(“flyway migration result, executed [{}] migrations on schema [{}] on database [{}]”, result.migrationsExecuted, result.schemaName, result.database);

Authentication & Security

Authentication Storing own passwords is not an option in a hobby project OAuth is the perfect fit to give away responsibilities Login via github, then storing in session Advantage: Unique usernames Solution: pac4j

Pac4j javalin-pac4j Support for OAuth, SAML, CAS, OpenID Connect, SPNEGO OAuth: GitHub, LinkedIn, Dropbox, Facebook, Google, Twitter, Wordpress (Generic for custom endpoints)

Security - Organization check User loggedInUser = ctx.sessionAttribute(“user”); if (loggedInUser != null) { if (ctx.pathParamMap().containsKey(“organization”)) { final String pathOrganization = ctx.pathParam(“organization”); boolean isUserPartOfOrganization = loggedInUser.getOrganizations().stream() .anyMatch(o -> o.getName().equals(pathOrganization)); if (isUserPartOfOrganization == false) { throw new NotFoundResponse(“…”); } final Organization organization = organizationDao.findByName(pathOrganization); ctx.sessionAttribute(“organization”, organization); } } Can you spot the issue?

Web framework

HTTP Server & Framework Javalin Written in Kotlin On top of Jetty Sync/Async APIs

Javalin

Javalin App Setup Big App class, that could be modularized DB setup DAOs Routes & Controllers Background job initialization

App Setup OrganizationDao organizationDao = jdbi.onDemand(OrganizationDao.class); UserDao userDao = jdbi.onDemand(UserDao.class); EventDao eventDao = jdbi.onDemand(EventDao.class); AnalyticsDao analyticsDao = jdbi.onDemand(AnalyticsDao.class); ContentResourceDao contentResourceDao = jdbi.onDemand(ContentResourceDao.class); TalkDao talkDao = jdbi.onDemand(TalkDao.class); SubmissionDao submissionDao = jdbi.onDemand(SubmissionDao.class); CommentDao commentDao = jdbi.onDemand(CommentDao.class); EventTagsDao eventTagsDao = jdbi.onDemand(EventTagsDao.class); InvitationDao invitationDao = jdbi.onDemand(InvitationDao.class);

App Setup ReportingController reportingController = new ReportingController(eventDao, analyticsDao, organizationDao); SearchController searchController = new SearchController(eventDao, talkDao); TalkController talkController = new TalkController(talkDao, eventDao, contentResourceDao, submissionDao); SettingsController settingsController = new SettingsController(userDao, invitationDao); Security security = new Security(organizationDao, userDao, invitationDao); InviteeController inviteeController = new InviteeController(invitationDao);

Javalin - Handler registration app.before(“/app/*”, security.checkUserSessionHandler()); app.get(“/app/{organization}/dashboard”, reportingController.dashboard()); app.get(“/app/{organization}/reporting”, reportingController.reportingEcharts()); app.get(“/app/{organization}/settings”, settingsController.settings()); app.post(“/app/{organization}/search”, searchController.search()); post(“/app/{organization}/event/{id}/submission/{talkId}/attendees”, eventController.insertAttendeeAnalytics());

Sample handler public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; }

Persistent sessions across restarts this.app = Javalin.create(config -> { … config.sessionHandler(this::sqlSessionHandler); });

Persistent sessions across restarts private SessionHandler sqlSessionHandler() { SessionHandler sessionHandler = new SessionHandler(); SessionCache sessionCache = new DefaultSessionCache(sessionHandler); DatabaseAdaptor databaseAdaptor = new DatabaseAdaptor(); databaseAdaptor.setDatasource(this.dataSource); JDBCSessionDataStoreFactory jdbcSessionDataStoreFactory = new JDBCSessionDataStoreFactory(); jdbcSessionDataStoreFactory.setDatabaseAdaptor(databaseAdaptor); sessionCache.setSessionDataStore(jdbcSessionDataStoreFactory.getSessionDataStore(sessionHandler)); sessionHandler.setSessionCache(sessionCache); sessionHandler.setHttpOnly(true); } return sessionHandler;

Templating Language JTE Fast Autocomplete in the IDE Custom tags Template precompilation possible Further optimizations to reduce GC/memory and increase throughput

JTE @import static utils.TemplateUtils.capitalize @param String organization @param Talk talk @param List<Event> events <div class=”col-span-1”> <label for=”type” class=”block text-sm font-medium text-gray-700 font-mono”>Type</label> <select id=”type” name=”type” class=”mt-1 text-lg pl-2 pr-8 shadow-sm focus:bg-yellow-100 focus:border-none”> @for(var type : ContentResource.ResourceType.values()) <option value=”${type.name()}”>${capitalize(type.name())}</option> @endfor </select> </div>

JTE Benchmarks

JTE - Precompiled templates // precompile JTE templates tasks.precompileJte { sourceDirectory = Paths.get(project.projectDir.absolutePath, “src”, “main”, “jte”) targetDirectory = Paths.get(project.projectDir.absolutePath, “jte-classes”) compilePath = sourceSets.main.runtimeClasspath contentType = ContentType.Html compileArgs = [ ‘-source’, ‘16’ ] }

Frontend See https://roadmap.sh/frontend

Frontend

plugins { id “com.github.node-gradle.node” version “3.1.0” } node { // current LTS release version = “14.17.1” download = true } // build css and put it into resources directory task build(type: NpmTask) { dependsOn npmInstall npmCommand = [‘run’, ‘build’] onlyIf { File destFile = file(“$projectDir/build/src/flipbit.css”) File sourceFile = file(“$projectDir/src/flipbit.css”) if (!destFile.exists()) { return true } } } return sourceFile.lastModified() > destFile.lastModified() task forceBuild(type: NpmTask) { dependsOn npmInstall npmCommand = [‘run’, ‘build’] }

Frontend: package.json { } “name”: “css”, “description”: “tooling to create a custom tailwind build”, “scripts”: { “build”: “./node_modules/.bin/tailwindcss —minify -i src/flipbit.css -o build/src/flipbit.css”, “watch”: “./node_modules/.bin/tailwindcss -w —minify -i src/flipbit.css -o build/src/flipbit.css” }, “devDependencies”: { “@tailwindcss/typography”: “0.4.1”, “@tailwindcss/forms”: “0.3.3”, “tailwindcss”: “2.2.7” }

Frontend: tailwind.config.js module.exports = { mode: ‘jit’, purge: [ ‘../src/main/jte//*.jte’, ‘./src//.html’, ‘./src/**/.js’, ], plugins: [ require(‘@tailwindcss/typography’), require(‘@tailwindcss/forms’) ] }

Frontend - fast builds

htmx - Ajax without JavaScript htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext htmx is small (~10k min.gz’d), dependency-free, extendable & IE11 compatible

htmx in HTML <div class=”hidden pt-3” id=”resource-form”> <form hx-post=”/app/${organization}/talk/${talk.getId()}/resource” hx-push-url=”true” hx-target=”#main”> … <input … /> <button type=”submit” class=”flipbit-button”> Save </button> </form> </div>

hyperscript hyperscript is an easy and approachable language designed for modern frontend web development

htmx + hyperscript = no javascript <button _=”on click toggle .hidden on #resource-form” class=”flipbit-button”>Add</button>

htmx - hide/show on click <button class=”flipbit-button” _=”on click hide me show #form-add-submissions”>

htmx - hide/show on hover <div _=”on mouseenter toggle .visible on #help until mouseleave”> Mouse Over Me! </div> <div id=”help”> I’m a helpful message!</div>

htmx - validate on change <input type=”date” name=”start_at” id=”start” class=”…” value=”${ZonedDateTime.now().format(YEAR_MONTH_DAY_FORMATTER)}” _=”on change if #end.value == ” or #start.value > #end.value then set { value: #start.value, min: #start.value } on #end” />

htmx - multiple events <div class=”px-8 relative inline-block”> <input class=”…” id=”searchbar” type=”text” placeholder=” Search…” name=”flipbit-query” hx-post=”/app/${organization}/search” hx-trigger=”keyup changed delay:500ms” hx-target=”#search-results” hx-indicator=”.htmx-indicator” _=”on keyup[key is ‘/’ and target does not match <input, textarea/>] from <body/> call me.focus() on blur wait 200ms hide #search-results on focus show #search-results”/> <div id=”search-results” class=”absolute z-50 w-64 mt-1 pl-4 flipbit-search-results”> </div> </div>

Development Environment & Testing

Local system setup via direnv direnv - unclutter your .profile .envrc dotenv

Local system setup via direnv .env GITHUB_KEY=”<<KEY>>” GITHUB_SECRET=”<<SECRET>>” GITHUB_CALLBACK_URL=”http://localhost:7000/auth/login/github” MODE=development PORT=”7000” YOUTUBE_TOKEN=”<<YT_TOKEN>>” #DB_USERNAME=”sa” #DB_PASSWORD=”” #DB_URL=”jdbc:h2:mem:flipbit_in_memory;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false” DB_USERNAME=”alr” DB_PASSWORD=”” DB_URL=”jdbc:postgresql://localhost:5432/alr”

Development vs. production mode public enum Mode { PRODUCTION, DEVELOPMENT; public static Mode current() { final String mode = System.getenv(“MODE”).toUpperCase(Locale.ROOT); return Mode.valueOf(mode); } public static boolean isProduction() { return current() == PRODUCTION; } } public static boolean isDevelopment() { return current() == DEVELOPMENT; }

Development vs. production mode // this is the location in the dockerfile // probably makes more sense to put this into an env var if (Mode.isProduction()) { config.addStaticFiles(“/css”, “/flipbit-app/resources/css”, Location.EXTERNAL); } else { config.addStaticFiles(“/css”, “build/resources/css”, Location.EXTERNAL); }

Development vs. production mode final TemplateEngine templateEngine; // no template compilation in prod, because of the read only docker image // templates are created as part of the build if (Mode.isProduction()) { templateEngine = TemplateEngine.createPrecompiled(Paths.get(“/jte-classes”), ContentType.Html); } else { final DirectoryCodeResolver resolver = new DirectoryCodeResolver(Paths.get(“src”, “main”, “jte”)); templateEngine = TemplateEngine.create(resolver, ContentType.Html); templateEngine.precompileAll(); } JavalinJte.configure(templateEngine);

Testing testImplementation “org.assertj:assertj-core:3.19.0” testImplementation ‘org.mockito:mockito-core:3.8.0’ testImplementation ‘org.junit-pioneer:junit-pioneer:1.3.8’ testImplementation “org.testcontainers:junit-jupiter:1.15.2” testImplementation “org.testcontainers:postgresql:1.15.2” testImplementation ‘nl.jqno.equalsverifier:equalsverifier:3.5.5’ testImplementation ‘org.junit.jupiter:junit-jupiter-api:5.7.1’ testRuntimeOnly ‘org.junit.jupiter:junit-jupiter-engine:5.7.1’

EqualsVerifier @Test public void testEqualsVerifierUser() { EqualsVerifier.simple().forClass(User.class).verify(); }

Supporting Tools Forbidden APIs Errorprone spotless

Errorprone

Forbidden APIS

Production environment

Packaging gradle application plugin Creates shell scripts for services automatically via installDist task bin/flipbit-app Package size: less than 13MB

Packaging: ZGC def jvmOptions = [“-Xmx512m”, “-XX:+UseZGC”] startScripts { defaultJvmOpts = jvmOptions } run { dependsOn(‘copyCssFile’) jvmArgs += jvmOptions }

Packaging: Dockerfile FROM gradle:7.0-jdk16 as build COPY —chown=gradle:gradle . /home/gradle/src WORKDIR /home/gradle/src RUN gradle clean check precompileJte installDist —no-daemon ############### FROM openjdk:16-slim-buster RUN addgroup —system flipbit && adduser —system flipbit —ingroup flipbit USER flipbit:flipbit WORKDIR /flipbit-app COPY —from=build /home/gradle/src/build/ /flipbit-app COPY —from=build /home/gradle/src/jte-classes/ /jte-classes EXPOSE 7000 ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]

Packaging: APM agent def agentVersion = “1.25.0” task downloadAgent(type: Download) { mkdir “.gradle/agent/” src “https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/${agentVersion}/elastic-apm-agent-${agentVersion}.jar” dest new File(“.gradle/agent”, “elastic-apm-agent-${agentVersion}.jar”) onlyIfModified true overwrite false } task copyAgent(type: Copy) { dependsOn ‘downloadAgent’ from(“.gradle/agent/elastic-apm-agent-${agentVersion}.jar”) { rename “(.*)-${agentVersion}.jar”, ‘$1.jar’ } into “build/install/flipbit-app/agent/” } copyAgent.mustRunAfter installDist ENV FLIPBIT_APP_OPTS=”-javaagent:/flipbit-app/install/flipbit-app/agent/elastic-apm-agent.jar” ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]

Deployment git push spinscale main Digital Ocean App Platform (Github hook building & deploying the Docker image) Postgresql Costs: 27$ per month (different kind of gym costs One click scale up )

Deployment alternatives Own VM: Price | Maintenance | Setup (TLS) Heroku: Qovery Digital Ocean App Platform Others: fly.io, render, AWS Beanstalk Java makes options more expensive | Custom rollout

Summary

But why no…? Kotlin/Node/Ruby? Didn’t want to learn a new language, getting results fast Spring Boot? Not increasing my productivity at all… SPA with a JSON API? Making some things easier, and others harder. Requires to learn Svelte/Vue, TypeScript, node ecosystem Quarkus/Micronaut Not comfortable enough yet, but a good candidate for the future Serverless Google Cloud Run is a good candidate, app must be stateless

The missing parts So much… all the time Development & Testing Build Metrics Ops Security UI

Missing: Development & Testing Build high level classes around low level DAOs Don’t run flyway for all unit tests, don’t recreate h2 db for every test class Unique incrementing ids per organization Easy local development: offline support (no oauth) HtmlUnit/E2E Testing H2/Postgres gap

Missing: Build GraalVM all the things - reduces monitoring Reproducible builds

Missing: Metrics Business metrics/health endpoints Analytics (not needed with web logs?) Monitoring/APM

Missing: Logs Logs is an issue on PaaS No stored logs in a storage bucket on DigitalOcean Storing logs can cost the same than running your app… Remote logging provider? Test when endpoint is not reachable

Missing: Ops Terraform setup for DO Rate Limiting (Javalin built-in?) Terraform based monitoring (for example Checkly) APM: Every agent supports Spring Boot and then nothing…

Missing: Security Add all the fancy security headers, CORS X-XXS-Protection , X-Frame-Options , Content-Security-Policy, X-Content-Type-Options . See also OWASP Secure Header Project Add seccomp policy to never fork processes Become smarter about landlock as a java security manager replacement

Missing: UI Rework UI to make it acceptable CSS only chart libraries like chartscss.org

Summary Don’t underestimate the work, but Keep it fun! There is no failure! Combining different libraries means a lot of learning htmx/hyperscript is a frontend revelation for backend developers PaaS > K8s

Standard System Radical Simplicity Browser App Radical simplicity Application API s Micro Service 1 Micro Service 2 Message Queue Micro Service 3

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale

Discussion What technologies would you use? Where did I go wrong?

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale