A presentation at Java User Group Munich by Alexander Reelsen
Using Java for a Greenfield Project From idea to implementation. All by yourself. Featuring boring technologies only! Alexander Reelsen alex@spinscale.de | @spinscale
Today’s goal How to run and maintain a modern web java application with minimal resources
About me By day: Community @ Elastic By night: sleep In between: Elasticsearch enthusiast, web framework evaluator, serverless observer, distributed work proponent, basketball fan Java since ‘07
Why greenfield? Assessing current skills & technologies Acquiring new skills & technologies Communication/Feedback Business idea Test out competition Understand the full application lifecycle Just for fun
Limitations Full-time job Family & Kids Covid-19 No full stack developer
Management summary Effectiveness - Do the right things Efficiency - Do the things right Efficiency requires effectiveness
Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes!
Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes!
Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes! Easy rollouts - git pushes No login handling - OAuth Optimize for (single) developer productivity Small deployments, few dependencies! Ability to debug
Idea Developer Advocate Conference & Talks Management
Conferences @Elastic Labels for tagging/reporting Milestone for CfP closure No fixed date fields No final closing report (pictures, sponsors, competitors) No easy overview over submitted talks
Stats (+Airtable) Participants Youtube links
Project Management Not needed, you got it all in your head, or not? Pen & Paper for initial modelling (including entities) Jetbrains Spaces, mainly as repo and kanban board
Boring technologies Javalin JDBI Pac4j Frontend: htmx, apache echarts, custom tailwind css build Database: Postgres/H2
But why no…? Kotlin/Node/Ruby? Didn’t want to learn a new language, getting results fast Spring Boot? Not increasing my productivity at all… SPA with a JSON API? Making some things easier, and others harder. Requires to learn Svelte/Vue, TypeScript, node ecosystem Quarkus/Micronaut Not comfortable enough yet, but a good candidate for the future Serverless Google Cloud Run is a good candidate, app must be stateless
HTTP Server & Framework Javalin Written in Kotlin On top of Jetty Sync/Async APIs Various Template Languages
Javalin
Templating Language JTE - aka return of the JSP Fast Autocomplete in the IDE Custom tags Template precompilation possible Further optimizations to reduce GC/memory and increase throughput
JTE @import static utils.TemplateUtils.capitalize @param String organization @param Talk talk @param List<Event> events <div class=”col-span-1”> <label for=”type” class=”block text-sm font-medium text-gray-700 font-mono”>Type</label> <select id=”type” name=”type” class=”mt-1 text-lg pl-2 pr-8 shadow-sm focus:bg-yellow-100 focus:border-none”> @for(var type : ContentResource.ResourceType.values()) <option value=”${type.name()}”>${capitalize(type.name())}</option> @endfor </select> </div>
Persistence Layer 1. Elasticsearch using templating to prevent Object Mapping and not take the whole of Elasticsearch as dependency into 2. Postgres/H2 using ebean 3. Postgres/H2 using JDBI
Database setup if (System.getenv(“DB_URL”).startsWith(“jdbc:h2:”)) { JdbcDataSource jdbcDataSource = new JdbcDataSource(); jdbcDataSource.setUser(System.getenv(“DB_USERNAME”)); jdbcDataSource.setPassword(System.getenv(“DB_PASSWORD”)); jdbcDataSource.setUrl(System.getenv(“DB_URL”)); this.dataSource = jdbcDataSource; this.jdbi = Jdbi.create(dataSource); this.jdbi.installPlugin(new H2DatabasePlugin()); } else {
} else { PGSimpleDataSource ds = new PGSimpleDataSource(); ds.setUser(System.getenv(“DB_USERNAME”)); ds.setPassword(System.getenv(“DB_PASSWORD”)); ds.setUrl(System.getenv(“DB_URL”)); HikariConfig hc = new HikariConfig(); hc.setDataSource(ds); hc.setMinimumIdle(1); hc.setMaximumPoolSize(6); final HikariDataSource dataSource = new HikariDataSource(hc); this.jdbi = Jdbi.create(dataSource); this.dataSource = dataSource; this.jdbi.installPlugin(new PostgresPlugin()); } this.jdbi.installPlugin(new SqlObjectPlugin());
JDBI Jdbi provides convenient, idiomatic access to relational data in Java.
JDBI Jdbi provides convenient, idiomatic access to relational data in Java. Unlike an ORM, we do not aim to provide a complete object relational mapping framework - instead of that hidden complexity, we provide building blocks that allow you to construct the mapping between relations and objects as appropriate for your application.
User entity public final class User { private Long id; … private List<Organization> organizations = new ArrayList<>(1); }
JDBI @RegisterBeanMapper(value = User.class, prefix = “u”) @RegisterBeanMapper(value = Organization.class, prefix = “o”) @UseRowReducer(UserOrganizationDocReducer.class) @SqlQuery(“”” SELECT u.id u_id, u.username u_username, u.display_name u_display_name, u.picture_url u_picture_url, u.email u_email, u.created_at u_created_at, u.modified_at u_modified_at, o.id o_id, o.name o_name, o.created_at o_created_at, o.modified_at o_modified_at FROM users u JOIN user_organizations uo ON uo.user_id = u.id JOIN organization o ON uo.organization_id = o.id WHERE u.username = ? “”“) User findByUsername(String username);
JDBI - RowReducer class UserOrganizationDocReducer implements LinkedHashMapRowReducer<Long, User> { @Override public void accumulate(Map<Long, User> map, RowView rv) { User u = map.computeIfAbsent(rv.getColumn(“u_id”, Long.class), id -> rv.getRow(User.class)); } } if (rv.getColumn(“o_id”, Long.class) != null) { u.getOrganizations().add(rv.getRow(Organization.class)); }
JDBI @SqlQuery(“”” SELECT to_char(DATE_TRUNC(‘month’,created_at), ‘YYYY-MM’) AS month, SUM(count) AS count FROM analytics WHERE organization_id = :organization.id AND created_at >= :from AND created_at <= :to AND type = :type GROUP BY month; “”“) @KeyColumn(“month”) @ValueColumn(“count”) Map<String, Long> sumAndGroupByMonth( @BindBean(“organization”) Organization organization, @Bind(“type”) Analytics.AnalyticsType type, @Bind(“from”) OffsetDateTime from, @Bind(“to”) OffsetDateTime to);
JDBI - enums public final class Comment { @EnumByName public enum CommentType { COMMENT, REPORT }
JDBI - security Multi-tenant system, implicit security issue How to prevent unauthorized access? Every DAO read operation requires an organization argument Organization cannot be injected from outside
Flyway Database Migrations // run the flyway migration before the app starts Flyway flyway = Flyway.configure().dataSource(dataSource).load(); final MigrateResult result = flyway.migrate(); logger.info(“flyway migration result, executed [{}] migrations on schema [{}] on database [{}]”, result.migrationsExecuted, result.schemaName, result.database);
Local system setup via direnv direnv - unclutter your .profile .envrc dotenv
Local system setup via direnv .env GITHUB_KEY=”<<KEY>>” GITHUB_SECRET=”<<SECRET>>” GITHUB_CALLBACK_URL=”http://localhost:7000/auth/login/github” MODE=development PORT=”7000” YOUTUBE_TOKEN=”<<YT_TOKEN>>” #DB_USERNAME=”sa” #DB_PASSWORD=”” #DB_URL=”jdbc:h2:mem:flipbit_in_memory;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false” DB_USERNAME=”alr” DB_PASSWORD=”” DB_URL=”jdbc:postgresql://localhost:5432/alr”
Development vs. production mode public enum Mode { PRODUCTION, DEVELOPMENT; public static Mode current() { final String mode = System.getenv(“MODE”).toUpperCase(Locale.ROOT); return Mode.valueOf(mode); } public static boolean isProduction() { return current() == PRODUCTION; } } public static boolean isDevelopment() { return current() == DEVELOPMENT; }
Development vs. production mode // this is the location in the dockerfile // probably makes more sense to put this into an env var if (Mode.isProduction()) { config.addStaticFiles(“/css”, “/flipbit-app/resources/css”, Location.EXTERNAL); } else { config.addStaticFiles(“/css”, “build/resources/css”, Location.EXTERNAL); }
Development vs. production mode final TemplateEngine templateEngine; // no template compilation in prod, because of the read only docker image, templates are created as part of the build if (Mode.isProduction()) { templateEngine = TemplateEngine.createPrecompiled(Paths.get(“/jte-classes”), ContentType.Html); } else { final DirectoryCodeResolver resolver = new DirectoryCodeResolver(Paths.get(“src”, “main”, “jte”)); templateEngine = TemplateEngine.create(resolver, ContentType.Html); templateEngine.precompileAll(); } JavalinJte.configure(templateEngine);
Precompiled templates // precompile JTE templates tasks.precompileJte { sourceDirectory = Paths.get(project.projectDir.absolutePath, “src”, “main”, “jte”) targetDirectory = Paths.get(project.projectDir.absolutePath, “jte-classes”) compilePath = sourceSets.main.runtimeClasspath contentType = ContentType.Html compileArgs = [ ‘-source’, ‘16’ ] }
Authentication Storing own passwords is not an option in a hobby project OAuth is the perfect fit to give away responsibilities Login via github, then storing in session Advantage: Unique usernames Solution: pac4j
Pac4j javalin-pac4j Support for OAuth, SAML, CAS, OpenID Connect, SPNEGO OAuth: GitHub, LinkedIn, Dropbox, Facebook, Google, Twitter, Wordpress (Generic for custom endpoints)
App Setup Big App class, that could be modularized DB setup DAOs Routes & Controllers Background job initialization
App Setup OrganizationDao organizationDao = jdbi.onDemand(OrganizationDao.class); UserDao userDao = jdbi.onDemand(UserDao.class); EventDao eventDao = jdbi.onDemand(EventDao.class); AnalyticsDao analyticsDao = jdbi.onDemand(AnalyticsDao.class); ContentResourceDao contentResourceDao = jdbi.onDemand(ContentResourceDao.class); TalkDao talkDao = jdbi.onDemand(TalkDao.class); SubmissionDao submissionDao = jdbi.onDemand(SubmissionDao.class); CommentDao commentDao = jdbi.onDemand(CommentDao.class); EventTagsDao eventTagsDao = jdbi.onDemand(EventTagsDao.class); InvitationDao invitationDao = jdbi.onDemand(InvitationDao.class);
App Setup ReportingController reportingController = new ReportingController(eventDao, analyticsDao, organizationDao); SearchController searchController = new SearchController(eventDao, talkDao); TalkController talkController = new TalkController(talkDao, eventDao, contentResourceDao, submissionDao); SettingsController settingsController = new SettingsController(userDao, invitationDao); Security security = new Security(organizationDao, userDao, invitationDao); InviteeController inviteeController = new InviteeController(invitationDao);
App Setup app.routes(() -> { // login button redirection get(“/auth/login/github”, security.callbackHandler()); // ensure that logout kills the session, that could be replaced with checkUserSessionHandler()? before(“/auth/logout”, security.securityHandler()); get(“/auth/logout”, security.logoutHandler()); // the main handler that ensures, everything behind /app is secured and we have a user session before(“/app/*”, security.checkUserSessionHandler()); } …
App setup get(“/app/:organization/dashboard”, reportingController.dashboard()); get(“/app/:organization/reporting”, reportingController.reportingEcharts()); get(“/app/:organization/settings”, settingsController.settings()); post(“/app/:organization/search”, searchController.search());
App setup get(“/app/:organization/event”, eventController.renderEvents()); get(“/app/:organization/event/new”, eventController.showCreateEventForm()); post(“/app/:organization/event”, eventController.createEvent()); get(“/app/:organization/event/:id”, eventController.renderEvent()); post(“/app/:organization/event/:id”, eventController.updateExistingEvent()); get(“/app/:organization/event/:id/edit”, eventController.editEventForm()); post(“/app/:organization/event/:id/delete”, eventController.deleteEvent()); post(“/app/:organization/event/:id/comment”, eventController.addComment()); post(“/app/:organization/event/:id/submission”, eventController.addSubmission()); post(“/app/:organization/event/:id/submission/:talkId/status/:status”, eventController.changeTalkStatus()); post(“/app/:organization/event/:id/submission/:talkId/attendees”, eventController.insertAttendeeAnalytics());
Sample controller public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; } Organization in session to ensure it cannot be modified Event deletion is based on organization to prevent deleting other’s events
Sample controller public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; } @SqlUpdate(“UPDATE events SET active = false, modified_at = NOW() WHERE id = :event_id AND organization_id = :organization_id”) void delete(@Bind(“event_id”) long eventId, @Bind(“organization_id”) long organizationId);
Organizational check User loggedInUser = ctx.sessionAttribute(“user”); if (loggedInUser != null) { if (ctx.pathParamMap().containsKey(“organization”)) { final String pathOrganization = ctx.pathParam(“organization”); boolean isUserPartOfOrganization = loggedInUser.getOrganizations().stream() .anyMatch(o -> o.getName().equals(pathOrganization)); if (isUserPartOfOrganization == false) { throw new NotFoundResponse(“…”); } final Organization organization = organizationDao.findByName(pathOrganization); ctx.sessionAttribute(“organization”, organization); } } Can you spot the issue?
Testing testImplementation “org.assertj:assertj-core:3.19.0” testImplementation ‘org.mockito:mockito-core:3.8.0’ testImplementation ‘org.junit-pioneer:junit-pioneer:1.3.8’ testImplementation “org.testcontainers:junit-jupiter:1.15.2” testImplementation “org.testcontainers:postgresql:1.15.2” testImplementation ‘nl.jqno.equalsverifier:equalsverifier:3.5.5’ testImplementation ‘org.junit.jupiter:junit-jupiter-api:5.7.1’ testRuntimeOnly ‘org.junit.jupiter:junit-jupiter-engine:5.7.1’
EqualsVerifier @Test public void testEqualsVerifierUser() { EqualsVerifier.simple().forClass(User.class).verify(); }
Supporting Tools Forbidden APIs Errorprone
Errorprone
Forbidden APIS
Frontend See https://roadmap.sh/frontend
Frontend
Frontend Custom Tailwind CSS build
Frontend Blog post + Sample repo
plugins { id “com.github.node-gradle.node” version “3.0.1” } node { // current LTS release version = “14.16.1” download = true } // build css and put it into resources directory task build(type: NpxTask) { dependsOn npmInstall command = ‘./node_modules/.bin/snowpack’ args = [‘build’] onlyIf { File destFile = file(“$projectDir/build/src/flipbit.css”) File sourceFile = file(“$projectDir/src/flipbit.css”) if (!destFile.exists()) { return true } } } return sourceFile.lastModified() > destFile.lastModified() task forceBuild(type: NpxTask) { dependsOn npmInstall command = ‘./node_modules/.bin/snowpack’ args = [‘build’] }
htmx - Ajax without JavaScript htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext htmx is small (~10k min.gz’d), dependency-free, extendable & IE11 compatible
htmx in HTML <div class=”hidden pt-3” id=”resource-form”> <form hx-post=”/app/${organization}/talk/${talk.getId()}/resource” hx-push-url=”true” hx-target=”#main”> … <input … /> <button type=”submit” class=”flipbit-button”> Save </button> </form> </div>
Server side handling public Handler addResource() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); long id = ctx.pathParam(“id”, Long.class).get(); Talk talk = talkDao.findByIdAndOrganization(id, organization); String url = URI.create(ctx.formParam(“url”)).toString(); ContentResource.ResourceType type = ContentResource.ResourceType.valueOf(ctx.formParam(“type”).toUpperCase(Locale.ROOT)); final ContentResource contentResource = new ContentResource(); contentResource.setType(type); contentResource.setUrl(url); contentResource.setOrganization(organization); contentResourceDao.insert(talk.getId(), contentResource); } }; ctx.redirect(“/app/” + organization.getName() + “/talk/” + talk.getId());
Rollout git push spinscale main Digital Ocean App Platform Postgresql Costs: 27$ per month (different kind of gym costs One click scale up )
Rollout alternatives Own VM: Price | Maintenance | Setup (TLS) Heroku: Qovery Digital Ocean App Platform Others: fly.io, AWS Beanstalk Java makes options more expensive | Custom rollout
Start scripts gradle application plugin Creates shell scripts for services automatically via installDist task bin/flipbit-app Package size: less than 15MB
Use ZGC def jvmOptions = [“-Xmx512m”, “-XX:+UseZGC”] startScripts { defaultJvmOpts = jvmOptions } run { dependsOn(‘copyCssFile’) jvmArgs += jvmOptions }
Docker image creation App Platform is read-only: Precompiled templates Shell script creation!
Dockerfile FROM gradle:7.0-jdk16 as build COPY —chown=gradle:gradle . /home/gradle/src WORKDIR /home/gradle/src RUN gradle clean check precompileJte installDist —no-daemon ############### FROM openjdk:16-slim-buster RUN addgroup —system flipbit && adduser —system flipbit —ingroup flipbit USER flipbit:flipbit WORKDIR /flipbit-app COPY —from=build /home/gradle/src/build/ /flipbit-app COPY —from=build /home/gradle/src/jte-classes/ /jte-classes EXPOSE 7000 ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]
Dockerfile - agent support // download the lightstep agent as part of our bundle // so we can have autoinstrumentation def agentVersion = “0.16.0” task downloadAgent(type: Download) { mkdir “.gradle/agent/” src “https://github.com/lightstep/otel-launcher-java/releases/download/${agentVersion}/lightstep-opentelemetry-ja vaagent-${agentVersion}.jar” dest new File(“.gradle/agent”, “lightstep-opentelemetry-agent-${agentVersion}.jar”) onlyIfModified true overwrite false } task copyAgent(type: Copy) { dependsOn ‘downloadAgent’ from(“.gradle/agent/lightstep-opentelemetry-agent-${agentVersion}.jar”) { rename “(.*)-${agentVersion}.jar”, ‘$1.jar’ } into “build/install/flipbit-app/agent/” } copyAgent.mustRunAfter installDist Download & copy agent via gradle Add env vars: LS_SERVICE_NAME & FLIPBIT_APP_OPTS
Persistent sessions across restarts this.app = Javalin.create(config -> { … config.sessionHandler(this::sqlSessionHandler); });
Persistent sessions across restarts private SessionHandler sqlSessionHandler() { SessionHandler sessionHandler = new SessionHandler(); SessionCache sessionCache = new DefaultSessionCache(sessionHandler); DatabaseAdaptor databaseAdaptor = new DatabaseAdaptor(); databaseAdaptor.setDatasource(dataSource); JDBCSessionDataStoreFactory jdbcSessionDataStoreFactory = new JDBCSessionDataStoreFactory(); jdbcSessionDataStoreFactory.setDatabaseAdaptor(databaseAdaptor); sessionCache.setSessionDataStore(jdbcSessionDataStoreFactory.getSessionDataStore(sessionHandler)); sessionHandler.setSessionCache(sessionCache); sessionHandler.setHttpOnly(true); } return sessionHandler;
The missing parts So much… all the time Development & Testing Build Metrics Ops Security UI
Missing: Development & Testing Build high level classes around low level DAOs E2E-Testing Don’t run flyway for all unit tests, don’t recreate h2 db for every test class Unique incrementing ids per organization Easy local development: offline support (no oauth) H2/Postgres gap
Build GraalVM all the things - reduces monitoring Reproducible builds
Missing: Metrics Business metrics/health endpoints Analytics (not needed with web logs?) Monitoring/APM
Missing: Ops Terraform setup for DO Rate Limiting (Javalin built-in?) Terraform based monitoring (for example Checkly) APM: Every agent supports Spring Boot and then nothing…
Missing: Security Add all the fancy security headers, CORS X-XXS-Protection , X-Frame-Options , Content-Security-Policy, X-Content-Type-Options . See also OWASP Secure Header Project Add seccomp policy to never fork processes Become smarter about landlock as a java security manager replacement
Missing: UI Rework UI to make it acceptable
Summary Don’t under estimate the work There is no failure! Keep it fun! Combining different libraries means a lot of learning Also, who wants to test? :-) PaaS > K8s
Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale
Discussion What technologies would you use? Where did I go wrong?
Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale
So, you finally took the decision to run your little side project and registered an own domain name? Turns out the project is not done then. This talk walks through the steps from creating an initial application, testing it properly and optimizing for your own developer experience and happiness. You want simplicity and speed over everything else in that setup, as you don’t speed a ton of time on this. We will take a Javalin application, add authentication, figuring out which persistence layer to use, implementing business logic, adding a custom CSS framework build and backend developer friendly frontend libraries to iterate fast and reason for every step.
Also, how to deploy this as stress free as possible in production will be covered and how to manage your project - I’m sure you’re thrilled to know, how to manage yourself. And of course, we will also cover blind spots and why Java was picked.
Technologies shown in this presentation: Javalin, JDBI, pac4j, JTE, Digital Ocean App Platform, Tailwind CSS, htmx, Apache Echarts, Docker, gradle, errorprone, forbidden APIs.
for free. You
can too.