Up and running with a Java based greenfield side project

A presentation at Herbstcampus in September 2021 in by Alexander Reelsen

Slide 1

Slide 1

Greenfield-Webapplikation mit Java im Ein-Person-Selbstversuch Alexander Reelsen alr@spinscale.de | @spinscale

Slide 2

Slide 2

Today’s goal How to run and maintain a modern web java application with minimal resources

Slide 3

Slide 3

About me By day: Community @ Elastic By night: sleep In between: Elasticsearch enthusiast, web framework evaluator, serverless observer, distributed work proponent, basketball fan Java since ‘07

Slide 4

Slide 4

Why greenfield? Assessing current skills & technologies Acquiring new skills & technologies Communication/Feedback Business idea Test out competition Understand the full application lifecycle Just for fun

Slide 5

Slide 5

Limitations Full-time job Family & Kids Covid-19 No full stack developer

Slide 6

Slide 6

Requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Slide 7

Slide 7

Requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Slide 8

Slide 8

Requirements Do the needful - but not more Operational simplicity - No Kubernetes! Easy rollouts - git pushes No login handling - OAuth Optimize for (single) developer productivity Small deployments, few dependencies! Ability to debug

Slide 9

Slide 9

Idea Developer Advocate Conference & Talks Management

Slide 10

Slide 10

Conferences @Elastic Labels for tagging/reporting Milestone for CfP closure No fixed date fields No final closing report (pictures, sponsors, competitors) No easy overview over submitted talks

Slide 11

Slide 11

Stats (+Airtable) Participants Youtube links

Slide 12

Slide 12

Slide 13

Slide 13

Slide 14

Slide 14

Slide 15

Slide 15

Slide 16

Slide 16

Slide 17

Slide 17

Slide 18

Slide 18

Topics Database & Persistence Authentication & Security Web framework Frontend Development Environment & Testing Production Environment & Deployment

Slide 19

Slide 19

Topics Database & Persistence (JDBI, Postgres, HikariCP) Authentication & Security (Pac4j) Web framework (Javalin) Frontend (htmx, hyperscript, Apache echarts, tailwindcss) Development Environment & Testing (direnv, testcontainers) Production Environment & Deployment

Slide 20

Slide 20

Database & Persistence Less magic, more SQL

Slide 21

Slide 21

Persistence Layer 1. Elasticsearch using templating to prevent Object Mapping and not take the whole of Elasticsearch as dependency into 2. Postgres/H2 using ebean 3. Postgres/H2 using JDBI

Slide 22

Slide 22

Persistence Layer: Connection Pool PGSimpleDataSource ds = new PGSimpleDataSource(); ds.setUser(System.getenv(“DB_USERNAME”)); ds.setPassword(System.getenv(“DB_PASSWORD”)); ds.setUrl(System.getenv(“DB_URL”)); HikariConfig hc = new HikariConfig(); hc.setDataSource(ds); hc.setMinimumIdle(1); hc.setMaximumPoolSize(6); this.dataSource = new HikariDataSource(hc); this.jdbi = Jdbi.create(dataSource); this.jdbi.installPlugin(new PostgresPlugin()); this.jdbi.installPlugin(new SqlObjectPlugin());

Slide 23

Slide 23

JDBI Jdbi provides convenient, idiomatic access to relational data in Java.

Slide 24

Slide 24

JDBI Jdbi provides convenient, idiomatic access to relational data in Java. Unlike an ORM, we do not aim to provide a complete object relational mapping framework - instead of that hidden complexity, we provide building blocks that allow you to construct the mapping between relations and objects as appropriate for your application.

Slide 25

Slide 25

User entity public final class User { private Long id; … private List<Organization> organizations = new ArrayList<>(1); }

Slide 26

Slide 26

JDBI - Dao interfaces @RegisterBeanMapper(value = User.class, prefix = “u”) @RegisterBeanMapper(value = Organization.class, prefix = “o”) @UseRowReducer(UserOrganizationDocReducer.class) @SqlQuery(“”” SELECT u.id u_id, u.username u_username, u.display_name u_display_name, u.picture_url u_picture_url, u.email u_email, u.created_at u_created_at, u.modified_at u_modified_at, o.id o_id, o.name o_name, o.created_at o_created_at, o.modified_at o_modified_at FROM users u JOIN user_organizations uo ON uo.user_id = u.id JOIN organization o ON uo.organization_id = o.id WHERE u.username = ? “”“) User findByUsername(String username);

Slide 27

Slide 27

JDBI - RowReducer class UserOrganizationDocReducer implements LinkedHashMapRowReducer<Long, User> { @Override public void accumulate(Map<Long, User> map, RowView rv) { User u = map.computeIfAbsent(rv.getColumn(“u_id”, Long.class), id -> rv.getRow(User.class)); } } if (rv.getColumn(“o_id”, Long.class) != null) { u.getOrganizations().add(rv.getRow(Organization.class)); }

Slide 28

Slide 28

JDBI - Custom SQL @SqlQuery(“”” SELECT to_char(DATE_TRUNC(‘month’,created_at), ‘YYYY-MM’) AS month, SUM(count) AS count FROM analytics WHERE organization_id = :organization.id AND created_at >= :from AND created_at <= :to AND type = :type GROUP BY month; “”“) @KeyColumn(“month”) @ValueColumn(“count”) Map<String, Long> sumAndGroupByMonth( @BindBean(“organization”) Organization organization, @Bind(“type”) Analytics.AnalyticsType type, @Bind(“from”) OffsetDateTime from, @Bind(“to”) OffsetDateTime to);

Slide 29

Slide 29

JDBI - enums public final class Comment { @EnumByName public enum CommentType { COMMENT, REPORT }

Slide 30

Slide 30

JDBI - security Multi-tenant system, implicit security issue How to prevent unauthorized access? Every DAO read operation requires an organization argument Organization cannot be injected from outside

Slide 31

Slide 31

Flyway Database Migrations // run the flyway migration before the app starts Flyway flyway = Flyway.configure().dataSource(dataSource).load(); final MigrateResult result = flyway.migrate(); logger.info(“flyway migration result, executed [{}] migrations on schema [{}] on database [{}]”, result.migrationsExecuted, result.schemaName, result.database);

Slide 32

Slide 32

Authentication & Security

Slide 33

Slide 33

Authentication Storing own passwords is not an option in a hobby project OAuth is the perfect fit to give away responsibilities Login via github, then storing in session Advantage: Unique usernames Solution: pac4j

Slide 34

Slide 34

Pac4j javalin-pac4j Support for OAuth, SAML, CAS, OpenID Connect, SPNEGO OAuth: GitHub, LinkedIn, Dropbox, Facebook, Google, Twitter, Wordpress (Generic for custom endpoints)

Slide 35

Slide 35

Security - Organization check User loggedInUser = ctx.sessionAttribute(“user”); if (loggedInUser != null) { if (ctx.pathParamMap().containsKey(“organization”)) { final String pathOrganization = ctx.pathParam(“organization”); boolean isUserPartOfOrganization = loggedInUser.getOrganizations().stream() .anyMatch(o -> o.getName().equals(pathOrganization)); if (isUserPartOfOrganization == false) { throw new NotFoundResponse(“…”); } final Organization organization = organizationDao.findByName(pathOrganization); ctx.sessionAttribute(“organization”, organization); } } Can you spot the issue?

Slide 36

Slide 36

Web framework

Slide 37

Slide 37

HTTP Server & Framework Javalin Written in Kotlin On top of Jetty Sync/Async APIs

Slide 38

Slide 38

Javalin

Slide 39

Slide 39

Javalin App Setup Big App class, that could be modularized DB setup DAOs Routes & Controllers Background job initialization

Slide 40

Slide 40

App Setup OrganizationDao organizationDao = jdbi.onDemand(OrganizationDao.class); UserDao userDao = jdbi.onDemand(UserDao.class); EventDao eventDao = jdbi.onDemand(EventDao.class); AnalyticsDao analyticsDao = jdbi.onDemand(AnalyticsDao.class); ContentResourceDao contentResourceDao = jdbi.onDemand(ContentResourceDao.class); TalkDao talkDao = jdbi.onDemand(TalkDao.class); SubmissionDao submissionDao = jdbi.onDemand(SubmissionDao.class); CommentDao commentDao = jdbi.onDemand(CommentDao.class); EventTagsDao eventTagsDao = jdbi.onDemand(EventTagsDao.class); InvitationDao invitationDao = jdbi.onDemand(InvitationDao.class);

Slide 41

Slide 41

App Setup ReportingController reportingController = new ReportingController(eventDao, analyticsDao, organizationDao); SearchController searchController = new SearchController(eventDao, talkDao); TalkController talkController = new TalkController(talkDao, eventDao, contentResourceDao, submissionDao); SettingsController settingsController = new SettingsController(userDao, invitationDao); Security security = new Security(organizationDao, userDao, invitationDao); InviteeController inviteeController = new InviteeController(invitationDao);

Slide 42

Slide 42

Javalin - Handler registration app.before(“/app/*”, security.checkUserSessionHandler()); app.get(“/app/{organization}/dashboard”, reportingController.dashboard()); app.get(“/app/{organization}/reporting”, reportingController.reportingEcharts()); app.get(“/app/{organization}/settings”, settingsController.settings()); app.post(“/app/{organization}/search”, searchController.search()); post(“/app/{organization}/event/{id}/submission/{talkId}/attendees”, eventController.insertAttendeeAnalytics());

Slide 43

Slide 43

Sample handler public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; }

Slide 44

Slide 44

Persistent sessions across restarts this.app = Javalin.create(config -> { … config.sessionHandler(this::sqlSessionHandler); });

Slide 45

Slide 45

Persistent sessions across restarts private SessionHandler sqlSessionHandler() { SessionHandler sessionHandler = new SessionHandler(); SessionCache sessionCache = new DefaultSessionCache(sessionHandler); DatabaseAdaptor databaseAdaptor = new DatabaseAdaptor(); databaseAdaptor.setDatasource(this.dataSource); JDBCSessionDataStoreFactory jdbcSessionDataStoreFactory = new JDBCSessionDataStoreFactory(); jdbcSessionDataStoreFactory.setDatabaseAdaptor(databaseAdaptor); sessionCache.setSessionDataStore(jdbcSessionDataStoreFactory.getSessionDataStore(sessionHandler)); sessionHandler.setSessionCache(sessionCache); sessionHandler.setHttpOnly(true); } return sessionHandler;

Slide 46

Slide 46

Templating Language JTE Fast Autocomplete in the IDE Custom tags Template precompilation possible Further optimizations to reduce GC/memory and increase throughput

Slide 47

Slide 47

JTE @import static utils.TemplateUtils.capitalize @param String organization @param Talk talk @param List<Event> events <div class=”col-span-1”> <label for=”type” class=”block text-sm font-medium text-gray-700 font-mono”>Type</label> <select id=”type” name=”type” class=”mt-1 text-lg pl-2 pr-8 shadow-sm focus:bg-yellow-100 focus:border-none”> @for(var type : ContentResource.ResourceType.values()) <option value=”${type.name()}”>${capitalize(type.name())}</option> @endfor </select> </div>

Slide 48

Slide 48

JTE Benchmarks

Slide 49

Slide 49

JTE - Precompiled templates // precompile JTE templates tasks.precompileJte { sourceDirectory = Paths.get(project.projectDir.absolutePath, “src”, “main”, “jte”) targetDirectory = Paths.get(project.projectDir.absolutePath, “jte-classes”) compilePath = sourceSets.main.runtimeClasspath contentType = ContentType.Html compileArgs = [ ‘-source’, ‘16’ ] }

Slide 50

Slide 50

Frontend See https://roadmap.sh/frontend

Slide 51

Slide 51

Frontend

Slide 52

Slide 52

plugins { id “com.github.node-gradle.node” version “3.1.0” } node { // current LTS release version = “14.17.1” download = true } // build css and put it into resources directory task build(type: NpmTask) { dependsOn npmInstall npmCommand = [‘run’, ‘build’] onlyIf { File destFile = file(“$projectDir/build/src/flipbit.css”) File sourceFile = file(“$projectDir/src/flipbit.css”) if (!destFile.exists()) { return true } } } return sourceFile.lastModified() > destFile.lastModified() task forceBuild(type: NpmTask) { dependsOn npmInstall npmCommand = [‘run’, ‘build’] }

Slide 53

Slide 53

Frontend: package.json { } “name”: “css”, “description”: “tooling to create a custom tailwind build”, “scripts”: { “build”: “./node_modules/.bin/tailwindcss —minify -i src/flipbit.css -o build/src/flipbit.css”, “watch”: “./node_modules/.bin/tailwindcss -w —minify -i src/flipbit.css -o build/src/flipbit.css” }, “devDependencies”: { “@tailwindcss/typography”: “0.4.1”, “@tailwindcss/forms”: “0.3.3”, “tailwindcss”: “2.2.7” }

Slide 54

Slide 54

Frontend: tailwind.config.js module.exports = { mode: ‘jit’, purge: [ ‘../src/main/jte//*.jte’, ‘./src//.html’, ‘./src/**/.js’, ], plugins: [ require(‘@tailwindcss/typography’), require(‘@tailwindcss/forms’) ] }

Slide 55

Slide 55

Frontend - fast builds

Slide 56

Slide 56

htmx - Ajax without JavaScript htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext htmx is small (~10k min.gz’d), dependency-free, extendable & IE11 compatible

Slide 57

Slide 57

htmx in HTML <div class=”hidden pt-3” id=”resource-form”> <form hx-post=”/app/${organization}/talk/${talk.getId()}/resource” hx-push-url=”true” hx-target=”#main”> … <input … /> <button type=”submit” class=”flipbit-button”> Save </button> </form> </div>

Slide 58

Slide 58

hyperscript hyperscript is an easy and approachable language designed for modern frontend web development

Slide 59

Slide 59

htmx + hyperscript = no javascript <button _=”on click toggle .hidden on #resource-form” class=”flipbit-button”>Add</button>

Slide 60

Slide 60

htmx - hide/show on click <button class=”flipbit-button” _=”on click hide me show #form-add-submissions”>

Slide 61

Slide 61

htmx - hide/show on hover <div _=”on mouseenter toggle .visible on #help until mouseleave”> Mouse Over Me! </div> <div id=”help”> I’m a helpful message!</div>

Slide 62

Slide 62

htmx - validate on change <input type=”date” name=”start_at” id=”start” class=”…” value=”${ZonedDateTime.now().format(YEAR_MONTH_DAY_FORMATTER)}” _=”on change if #end.value == ” or #start.value > #end.value then set { value: #start.value, min: #start.value } on #end” />

Slide 63

Slide 63

htmx - multiple events <div class=”px-8 relative inline-block”> <input class=”…” id=”searchbar” type=”text” placeholder=” Search…” name=”flipbit-query” hx-post=”/app/${organization}/search” hx-trigger=”keyup changed delay:500ms” hx-target=”#search-results” hx-indicator=”.htmx-indicator” _=”on keyup[key is ‘/’ and target does not match <input, textarea/>] from <body/> call me.focus() on blur wait 200ms hide #search-results on focus show #search-results”/> <div id=”search-results” class=”absolute z-50 w-64 mt-1 pl-4 flipbit-search-results”> </div> </div>

Slide 64

Slide 64

Development Environment & Testing

Slide 65

Slide 65

Local system setup via direnv direnv - unclutter your .profile .envrc dotenv

Slide 66

Slide 66

Local system setup via direnv .env GITHUB_KEY=”<<KEY>>” GITHUB_SECRET=”<<SECRET>>” GITHUB_CALLBACK_URL=”http://localhost:7000/auth/login/github” MODE=development PORT=”7000” YOUTUBE_TOKEN=”<<YT_TOKEN>>” #DB_USERNAME=”sa” #DB_PASSWORD=”” #DB_URL=”jdbc:h2:mem:flipbit_in_memory;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false” DB_USERNAME=”alr” DB_PASSWORD=”” DB_URL=”jdbc:postgresql://localhost:5432/alr”

Slide 67

Slide 67

Development vs. production mode public enum Mode { PRODUCTION, DEVELOPMENT; public static Mode current() { final String mode = System.getenv(“MODE”).toUpperCase(Locale.ROOT); return Mode.valueOf(mode); } public static boolean isProduction() { return current() == PRODUCTION; } } public static boolean isDevelopment() { return current() == DEVELOPMENT; }

Slide 68

Slide 68

Development vs. production mode // this is the location in the dockerfile // probably makes more sense to put this into an env var if (Mode.isProduction()) { config.addStaticFiles(“/css”, “/flipbit-app/resources/css”, Location.EXTERNAL); } else { config.addStaticFiles(“/css”, “build/resources/css”, Location.EXTERNAL); }

Slide 69

Slide 69

Development vs. production mode final TemplateEngine templateEngine; // no template compilation in prod, because of the read only docker image // templates are created as part of the build if (Mode.isProduction()) { templateEngine = TemplateEngine.createPrecompiled(Paths.get(“/jte-classes”), ContentType.Html); } else { final DirectoryCodeResolver resolver = new DirectoryCodeResolver(Paths.get(“src”, “main”, “jte”)); templateEngine = TemplateEngine.create(resolver, ContentType.Html); templateEngine.precompileAll(); } JavalinJte.configure(templateEngine);

Slide 70

Slide 70

Testing testImplementation “org.assertj:assertj-core:3.19.0” testImplementation ‘org.mockito:mockito-core:3.8.0’ testImplementation ‘org.junit-pioneer:junit-pioneer:1.3.8’ testImplementation “org.testcontainers:junit-jupiter:1.15.2” testImplementation “org.testcontainers:postgresql:1.15.2” testImplementation ‘nl.jqno.equalsverifier:equalsverifier:3.5.5’ testImplementation ‘org.junit.jupiter:junit-jupiter-api:5.7.1’ testRuntimeOnly ‘org.junit.jupiter:junit-jupiter-engine:5.7.1’

Slide 71

Slide 71

EqualsVerifier @Test public void testEqualsVerifierUser() { EqualsVerifier.simple().forClass(User.class).verify(); }

Slide 72

Slide 72

Supporting Tools Forbidden APIs Errorprone spotless

Slide 73

Slide 73

Errorprone

Slide 74

Slide 74

Forbidden APIS

Slide 75

Slide 75

Production environment

Slide 76

Slide 76

Packaging gradle application plugin Creates shell scripts for services automatically via installDist task bin/flipbit-app Package size: less than 13MB

Slide 77

Slide 77

Packaging: ZGC def jvmOptions = [“-Xmx512m”, “-XX:+UseZGC”] startScripts { defaultJvmOpts = jvmOptions } run { dependsOn(‘copyCssFile’) jvmArgs += jvmOptions }

Slide 78

Slide 78

Packaging: Dockerfile FROM gradle:7.0-jdk16 as build COPY —chown=gradle:gradle . /home/gradle/src WORKDIR /home/gradle/src RUN gradle clean check precompileJte installDist —no-daemon ############### FROM openjdk:16-slim-buster RUN addgroup —system flipbit && adduser —system flipbit —ingroup flipbit USER flipbit:flipbit WORKDIR /flipbit-app COPY —from=build /home/gradle/src/build/ /flipbit-app COPY —from=build /home/gradle/src/jte-classes/ /jte-classes EXPOSE 7000 ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]

Slide 79

Slide 79

Packaging: APM agent def agentVersion = “1.25.0” task downloadAgent(type: Download) { mkdir “.gradle/agent/” src “https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/${agentVersion}/elastic-apm-agent-${agentVersion}.jar” dest new File(“.gradle/agent”, “elastic-apm-agent-${agentVersion}.jar”) onlyIfModified true overwrite false } task copyAgent(type: Copy) { dependsOn ‘downloadAgent’ from(“.gradle/agent/elastic-apm-agent-${agentVersion}.jar”) { rename “(.*)-${agentVersion}.jar”, ‘$1.jar’ } into “build/install/flipbit-app/agent/” } copyAgent.mustRunAfter installDist ENV FLIPBIT_APP_OPTS=”-javaagent:/flipbit-app/install/flipbit-app/agent/elastic-apm-agent.jar” ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]

Slide 80

Slide 80

Deployment git push spinscale main Digital Ocean App Platform (Github hook building & deploying the Docker image) Postgresql Costs: 27$ per month (different kind of gym costs One click scale up )

Slide 81

Slide 81

Deployment alternatives Own VM: Price | Maintenance | Setup (TLS) Heroku: Qovery Digital Ocean App Platform Others: fly.io, render, AWS Beanstalk Java makes options more expensive | Custom rollout

Slide 82

Slide 82

Summary

Slide 83

Slide 83

But why no…? Kotlin/Node/Ruby? Didn’t want to learn a new language, getting results fast Spring Boot? Not increasing my productivity at all… SPA with a JSON API? Making some things easier, and others harder. Requires to learn Svelte/Vue, TypeScript, node ecosystem Quarkus/Micronaut Not comfortable enough yet, but a good candidate for the future Serverless Google Cloud Run is a good candidate, app must be stateless

Slide 84

Slide 84

The missing parts So much… all the time Development & Testing Build Metrics Ops Security UI

Slide 85

Slide 85

Missing: Development & Testing Build high level classes around low level DAOs Don’t run flyway for all unit tests, don’t recreate h2 db for every test class Unique incrementing ids per organization Easy local development: offline support (no oauth) HtmlUnit/E2E Testing H2/Postgres gap

Slide 86

Slide 86

Missing: Build GraalVM all the things - reduces monitoring Reproducible builds

Slide 87

Slide 87

Missing: Metrics Business metrics/health endpoints Analytics (not needed with web logs?) Monitoring/APM

Slide 88

Slide 88

Missing: Logs Logs is an issue on PaaS No stored logs in a storage bucket on DigitalOcean Storing logs can cost the same than running your app… Remote logging provider? Test when endpoint is not reachable

Slide 89

Slide 89

Missing: Ops Terraform setup for DO Rate Limiting (Javalin built-in?) Terraform based monitoring (for example Checkly) APM: Every agent supports Spring Boot and then nothing…

Slide 90

Slide 90

Slide 91

Slide 91

Slide 92

Slide 92

Missing: Security Add all the fancy security headers, CORS X-XXS-Protection , X-Frame-Options , Content-Security-Policy, X-Content-Type-Options . See also OWASP Secure Header Project Add seccomp policy to never fork processes Become smarter about landlock as a java security manager replacement

Slide 93

Slide 93

Slide 94

Slide 94

Missing: UI Rework UI to make it acceptable CSS only chart libraries like chartscss.org

Slide 95

Slide 95

Summary Don’t underestimate the work, but Keep it fun! There is no failure! Combining different libraries means a lot of learning htmx/hyperscript is a frontend revelation for backend developers PaaS > K8s

Slide 96

Slide 96

Standard System Radical Simplicity Browser App Radical simplicity Application API s Micro Service 1 Micro Service 2 Message Queue Micro Service 3

Slide 97

Slide 97

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale

Slide 98

Slide 98

Discussion What technologies would you use? Where did I go wrong?

Slide 99

Slide 99

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale