Up and running with a Java based greenfield side project

A presentation at Java User Group Munich in May 2021 in by Alexander Reelsen

Slide 1

Slide 1

Using Java for a Greenfield Project From idea to implementation. All by yourself. Featuring boring technologies only! Alexander Reelsen alex@spinscale.de | @spinscale

Slide 2

Slide 2

Today’s goal How to run and maintain a modern web java application with minimal resources

Slide 3

Slide 3

About me By day: Community @ Elastic By night: sleep In between: Elasticsearch enthusiast, web framework evaluator, serverless observer, distributed work proponent, basketball fan Java since ‘07

Slide 4

Slide 4

Why greenfield? Assessing current skills & technologies Acquiring new skills & technologies Communication/Feedback Business idea Test out competition Understand the full application lifecycle Just for fun

Slide 5

Slide 5

Limitations Full-time job Family & Kids Covid-19 No full stack developer

Slide 6

Slide 6

Management summary Effectiveness - Do the right things Efficiency - Do the things right Efficiency requires effectiveness

Slide 7

Slide 7

Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Slide 8

Slide 8

Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes!

Slide 9

Slide 9

Operational requirements Do the needful - but not more Operational simplicity - No Kubernetes! Easy rollouts - git pushes No login handling - OAuth Optimize for (single) developer productivity Small deployments, few dependencies! Ability to debug

Slide 10

Slide 10

Idea Developer Advocate Conference & Talks Management

Slide 11

Slide 11

Conferences @Elastic Labels for tagging/reporting Milestone for CfP closure No fixed date fields No final closing report (pictures, sponsors, competitors) No easy overview over submitted talks

Slide 12

Slide 12

Stats (+Airtable) Participants Youtube links

Slide 13

Slide 13

Project Management Not needed, you got it all in your head, or not? Pen & Paper for initial modelling (including entities) Jetbrains Spaces, mainly as repo and kanban board

Slide 14

Slide 14

Boring technologies Javalin JDBI Pac4j Frontend: htmx, apache echarts, custom tailwind css build Database: Postgres/H2

Slide 15

Slide 15

But why no…? Kotlin/Node/Ruby? Didn’t want to learn a new language, getting results fast Spring Boot? Not increasing my productivity at all… SPA with a JSON API? Making some things easier, and others harder. Requires to learn Svelte/Vue, TypeScript, node ecosystem Quarkus/Micronaut Not comfortable enough yet, but a good candidate for the future Serverless Google Cloud Run is a good candidate, app must be stateless

Slide 16

Slide 16

HTTP Server & Framework Javalin Written in Kotlin On top of Jetty Sync/Async APIs Various Template Languages

Slide 17

Slide 17

Javalin

Slide 18

Slide 18

Templating Language JTE - aka return of the JSP Fast Autocomplete in the IDE Custom tags Template precompilation possible Further optimizations to reduce GC/memory and increase throughput

Slide 19

Slide 19

JTE @import static utils.TemplateUtils.capitalize @param String organization @param Talk talk @param List<Event> events <div class=”col-span-1”> <label for=”type” class=”block text-sm font-medium text-gray-700 font-mono”>Type</label> <select id=”type” name=”type” class=”mt-1 text-lg pl-2 pr-8 shadow-sm focus:bg-yellow-100 focus:border-none”> @for(var type : ContentResource.ResourceType.values()) <option value=”${type.name()}”>${capitalize(type.name())}</option> @endfor </select> </div>

Slide 20

Slide 20

Persistence Layer 1. Elasticsearch using templating to prevent Object Mapping and not take the whole of Elasticsearch as dependency into 2. Postgres/H2 using ebean 3. Postgres/H2 using JDBI

Slide 21

Slide 21

Database setup if (System.getenv(“DB_URL”).startsWith(“jdbc:h2:”)) { JdbcDataSource jdbcDataSource = new JdbcDataSource(); jdbcDataSource.setUser(System.getenv(“DB_USERNAME”)); jdbcDataSource.setPassword(System.getenv(“DB_PASSWORD”)); jdbcDataSource.setUrl(System.getenv(“DB_URL”)); this.dataSource = jdbcDataSource; this.jdbi = Jdbi.create(dataSource); this.jdbi.installPlugin(new H2DatabasePlugin()); } else {

Slide 22

Slide 22

} else { PGSimpleDataSource ds = new PGSimpleDataSource(); ds.setUser(System.getenv(“DB_USERNAME”)); ds.setPassword(System.getenv(“DB_PASSWORD”)); ds.setUrl(System.getenv(“DB_URL”)); HikariConfig hc = new HikariConfig(); hc.setDataSource(ds); hc.setMinimumIdle(1); hc.setMaximumPoolSize(6); final HikariDataSource dataSource = new HikariDataSource(hc); this.jdbi = Jdbi.create(dataSource); this.dataSource = dataSource; this.jdbi.installPlugin(new PostgresPlugin()); } this.jdbi.installPlugin(new SqlObjectPlugin());

Slide 23

Slide 23

JDBI Jdbi provides convenient, idiomatic access to relational data in Java.

Slide 24

Slide 24

JDBI Jdbi provides convenient, idiomatic access to relational data in Java. Unlike an ORM, we do not aim to provide a complete object relational mapping framework - instead of that hidden complexity, we provide building blocks that allow you to construct the mapping between relations and objects as appropriate for your application.

Slide 25

Slide 25

User entity public final class User { private Long id; … private List<Organization> organizations = new ArrayList<>(1); }

Slide 26

Slide 26

JDBI @RegisterBeanMapper(value = User.class, prefix = “u”) @RegisterBeanMapper(value = Organization.class, prefix = “o”) @UseRowReducer(UserOrganizationDocReducer.class) @SqlQuery(“”” SELECT u.id u_id, u.username u_username, u.display_name u_display_name, u.picture_url u_picture_url, u.email u_email, u.created_at u_created_at, u.modified_at u_modified_at, o.id o_id, o.name o_name, o.created_at o_created_at, o.modified_at o_modified_at FROM users u JOIN user_organizations uo ON uo.user_id = u.id JOIN organization o ON uo.organization_id = o.id WHERE u.username = ? “”“) User findByUsername(String username);

Slide 27

Slide 27

JDBI - RowReducer class UserOrganizationDocReducer implements LinkedHashMapRowReducer<Long, User> { @Override public void accumulate(Map<Long, User> map, RowView rv) { User u = map.computeIfAbsent(rv.getColumn(“u_id”, Long.class), id -> rv.getRow(User.class)); } } if (rv.getColumn(“o_id”, Long.class) != null) { u.getOrganizations().add(rv.getRow(Organization.class)); }

Slide 28

Slide 28

JDBI @SqlQuery(“”” SELECT to_char(DATE_TRUNC(‘month’,created_at), ‘YYYY-MM’) AS month, SUM(count) AS count FROM analytics WHERE organization_id = :organization.id AND created_at >= :from AND created_at <= :to AND type = :type GROUP BY month; “”“) @KeyColumn(“month”) @ValueColumn(“count”) Map<String, Long> sumAndGroupByMonth( @BindBean(“organization”) Organization organization, @Bind(“type”) Analytics.AnalyticsType type, @Bind(“from”) OffsetDateTime from, @Bind(“to”) OffsetDateTime to);

Slide 29

Slide 29

JDBI - enums public final class Comment { @EnumByName public enum CommentType { COMMENT, REPORT }

Slide 30

Slide 30

JDBI - security Multi-tenant system, implicit security issue How to prevent unauthorized access? Every DAO read operation requires an organization argument Organization cannot be injected from outside

Slide 31

Slide 31

Flyway Database Migrations // run the flyway migration before the app starts Flyway flyway = Flyway.configure().dataSource(dataSource).load(); final MigrateResult result = flyway.migrate(); logger.info(“flyway migration result, executed [{}] migrations on schema [{}] on database [{}]”, result.migrationsExecuted, result.schemaName, result.database);

Slide 32

Slide 32

Local system setup via direnv direnv - unclutter your .profile .envrc dotenv

Slide 33

Slide 33

Local system setup via direnv .env GITHUB_KEY=”<<KEY>>” GITHUB_SECRET=”<<SECRET>>” GITHUB_CALLBACK_URL=”http://localhost:7000/auth/login/github” MODE=development PORT=”7000” YOUTUBE_TOKEN=”<<YT_TOKEN>>” #DB_USERNAME=”sa” #DB_PASSWORD=”” #DB_URL=”jdbc:h2:mem:flipbit_in_memory;MODE=PostgreSQL;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false” DB_USERNAME=”alr” DB_PASSWORD=”” DB_URL=”jdbc:postgresql://localhost:5432/alr”

Slide 34

Slide 34

Development vs. production mode public enum Mode { PRODUCTION, DEVELOPMENT; public static Mode current() { final String mode = System.getenv(“MODE”).toUpperCase(Locale.ROOT); return Mode.valueOf(mode); } public static boolean isProduction() { return current() == PRODUCTION; } } public static boolean isDevelopment() { return current() == DEVELOPMENT; }

Slide 35

Slide 35

Development vs. production mode // this is the location in the dockerfile // probably makes more sense to put this into an env var if (Mode.isProduction()) { config.addStaticFiles(“/css”, “/flipbit-app/resources/css”, Location.EXTERNAL); } else { config.addStaticFiles(“/css”, “build/resources/css”, Location.EXTERNAL); }

Slide 36

Slide 36

Development vs. production mode final TemplateEngine templateEngine; // no template compilation in prod, because of the read only docker image, templates are created as part of the build if (Mode.isProduction()) { templateEngine = TemplateEngine.createPrecompiled(Paths.get(“/jte-classes”), ContentType.Html); } else { final DirectoryCodeResolver resolver = new DirectoryCodeResolver(Paths.get(“src”, “main”, “jte”)); templateEngine = TemplateEngine.create(resolver, ContentType.Html); templateEngine.precompileAll(); } JavalinJte.configure(templateEngine);

Slide 37

Slide 37

Precompiled templates // precompile JTE templates tasks.precompileJte { sourceDirectory = Paths.get(project.projectDir.absolutePath, “src”, “main”, “jte”) targetDirectory = Paths.get(project.projectDir.absolutePath, “jte-classes”) compilePath = sourceSets.main.runtimeClasspath contentType = ContentType.Html compileArgs = [ ‘-source’, ‘16’ ] }

Slide 38

Slide 38

Authentication Storing own passwords is not an option in a hobby project OAuth is the perfect fit to give away responsibilities Login via github, then storing in session Advantage: Unique usernames Solution: pac4j

Slide 39

Slide 39

Pac4j javalin-pac4j Support for OAuth, SAML, CAS, OpenID Connect, SPNEGO OAuth: GitHub, LinkedIn, Dropbox, Facebook, Google, Twitter, Wordpress (Generic for custom endpoints)

Slide 40

Slide 40

App Setup Big App class, that could be modularized DB setup DAOs Routes & Controllers Background job initialization

Slide 41

Slide 41

App Setup OrganizationDao organizationDao = jdbi.onDemand(OrganizationDao.class); UserDao userDao = jdbi.onDemand(UserDao.class); EventDao eventDao = jdbi.onDemand(EventDao.class); AnalyticsDao analyticsDao = jdbi.onDemand(AnalyticsDao.class); ContentResourceDao contentResourceDao = jdbi.onDemand(ContentResourceDao.class); TalkDao talkDao = jdbi.onDemand(TalkDao.class); SubmissionDao submissionDao = jdbi.onDemand(SubmissionDao.class); CommentDao commentDao = jdbi.onDemand(CommentDao.class); EventTagsDao eventTagsDao = jdbi.onDemand(EventTagsDao.class); InvitationDao invitationDao = jdbi.onDemand(InvitationDao.class);

Slide 42

Slide 42

App Setup ReportingController reportingController = new ReportingController(eventDao, analyticsDao, organizationDao); SearchController searchController = new SearchController(eventDao, talkDao); TalkController talkController = new TalkController(talkDao, eventDao, contentResourceDao, submissionDao); SettingsController settingsController = new SettingsController(userDao, invitationDao); Security security = new Security(organizationDao, userDao, invitationDao); InviteeController inviteeController = new InviteeController(invitationDao);

Slide 43

Slide 43

App Setup app.routes(() -> { // login button redirection get(“/auth/login/github”, security.callbackHandler()); // ensure that logout kills the session, that could be replaced with checkUserSessionHandler()? before(“/auth/logout”, security.securityHandler()); get(“/auth/logout”, security.logoutHandler()); // the main handler that ensures, everything behind /app is secured and we have a user session before(“/app/*”, security.checkUserSessionHandler()); } …

Slide 44

Slide 44

App setup get(“/app/:organization/dashboard”, reportingController.dashboard()); get(“/app/:organization/reporting”, reportingController.reportingEcharts()); get(“/app/:organization/settings”, settingsController.settings()); post(“/app/:organization/search”, searchController.search());

Slide 45

Slide 45

App setup get(“/app/:organization/event”, eventController.renderEvents()); get(“/app/:organization/event/new”, eventController.showCreateEventForm()); post(“/app/:organization/event”, eventController.createEvent()); get(“/app/:organization/event/:id”, eventController.renderEvent()); post(“/app/:organization/event/:id”, eventController.updateExistingEvent()); get(“/app/:organization/event/:id/edit”, eventController.editEventForm()); post(“/app/:organization/event/:id/delete”, eventController.deleteEvent()); post(“/app/:organization/event/:id/comment”, eventController.addComment()); post(“/app/:organization/event/:id/submission”, eventController.addSubmission()); post(“/app/:organization/event/:id/submission/:talkId/status/:status”, eventController.changeTalkStatus()); post(“/app/:organization/event/:id/submission/:talkId/attendees”, eventController.insertAttendeeAnalytics());

Slide 46

Slide 46

Sample controller public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; } Organization in session to ensure it cannot be modified Event deletion is based on organization to prevent deleting other’s events

Slide 47

Slide 47

Sample controller public Handler deleteEvent() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); final Long id = ctx.pathParam(“id”, Long.class).get(); eventDao.delete(id, organization.getId()); ctx.redirect(“/app/” + organization.getName() + “/event/”); }; } @SqlUpdate(“UPDATE events SET active = false, modified_at = NOW() WHERE id = :event_id AND organization_id = :organization_id”) void delete(@Bind(“event_id”) long eventId, @Bind(“organization_id”) long organizationId);

Slide 48

Slide 48

Organizational check User loggedInUser = ctx.sessionAttribute(“user”); if (loggedInUser != null) { if (ctx.pathParamMap().containsKey(“organization”)) { final String pathOrganization = ctx.pathParam(“organization”); boolean isUserPartOfOrganization = loggedInUser.getOrganizations().stream() .anyMatch(o -> o.getName().equals(pathOrganization)); if (isUserPartOfOrganization == false) { throw new NotFoundResponse(“…”); } final Organization organization = organizationDao.findByName(pathOrganization); ctx.sessionAttribute(“organization”, organization); } } Can you spot the issue?

Slide 49

Slide 49

Testing testImplementation “org.assertj:assertj-core:3.19.0” testImplementation ‘org.mockito:mockito-core:3.8.0’ testImplementation ‘org.junit-pioneer:junit-pioneer:1.3.8’ testImplementation “org.testcontainers:junit-jupiter:1.15.2” testImplementation “org.testcontainers:postgresql:1.15.2” testImplementation ‘nl.jqno.equalsverifier:equalsverifier:3.5.5’ testImplementation ‘org.junit.jupiter:junit-jupiter-api:5.7.1’ testRuntimeOnly ‘org.junit.jupiter:junit-jupiter-engine:5.7.1’

Slide 50

Slide 50

EqualsVerifier @Test public void testEqualsVerifierUser() { EqualsVerifier.simple().forClass(User.class).verify(); }

Slide 51

Slide 51

Supporting Tools Forbidden APIs Errorprone

Slide 52

Slide 52

Errorprone

Slide 53

Slide 53

Forbidden APIS

Slide 54

Slide 54

Frontend See https://roadmap.sh/frontend

Slide 55

Slide 55

Frontend

Slide 56

Slide 56

Frontend Custom Tailwind CSS build

Slide 57

Slide 57

Frontend Blog post + Sample repo

Slide 58

Slide 58

plugins { id “com.github.node-gradle.node” version “3.0.1” } node { // current LTS release version = “14.16.1” download = true } // build css and put it into resources directory task build(type: NpxTask) { dependsOn npmInstall command = ‘./node_modules/.bin/snowpack’ args = [‘build’] onlyIf { File destFile = file(“$projectDir/build/src/flipbit.css”) File sourceFile = file(“$projectDir/src/flipbit.css”) if (!destFile.exists()) { return true } } } return sourceFile.lastModified() > destFile.lastModified() task forceBuild(type: NpxTask) { dependsOn npmInstall command = ‘./node_modules/.bin/snowpack’ args = [‘build’] }

Slide 59

Slide 59

htmx - Ajax without JavaScript htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext htmx is small (~10k min.gz’d), dependency-free, extendable & IE11 compatible

Slide 60

Slide 60

htmx in HTML <div class=”hidden pt-3” id=”resource-form”> <form hx-post=”/app/${organization}/talk/${talk.getId()}/resource” hx-push-url=”true” hx-target=”#main”> … <input … /> <button type=”submit” class=”flipbit-button”> Save </button> </form> </div>

Slide 61

Slide 61

Server side handling public Handler addResource() { return ctx -> { Organization organization = ctx.sessionAttribute(“organization”); long id = ctx.pathParam(“id”, Long.class).get(); Talk talk = talkDao.findByIdAndOrganization(id, organization); String url = URI.create(ctx.formParam(“url”)).toString(); ContentResource.ResourceType type = ContentResource.ResourceType.valueOf(ctx.formParam(“type”).toUpperCase(Locale.ROOT)); final ContentResource contentResource = new ContentResource(); contentResource.setType(type); contentResource.setUrl(url); contentResource.setOrganization(organization); contentResourceDao.insert(talk.getId(), contentResource); } }; ctx.redirect(“/app/” + organization.getName() + “/talk/” + talk.getId());

Slide 62

Slide 62

Rollout git push spinscale main Digital Ocean App Platform Postgresql Costs: 27$ per month (different kind of gym costs One click scale up )

Slide 63

Slide 63

Rollout alternatives Own VM: Price | Maintenance | Setup (TLS) Heroku: Qovery Digital Ocean App Platform Others: fly.io, AWS Beanstalk Java makes options more expensive | Custom rollout

Slide 64

Slide 64

Start scripts gradle application plugin Creates shell scripts for services automatically via installDist task bin/flipbit-app Package size: less than 15MB

Slide 65

Slide 65

Use ZGC def jvmOptions = [“-Xmx512m”, “-XX:+UseZGC”] startScripts { defaultJvmOpts = jvmOptions } run { dependsOn(‘copyCssFile’) jvmArgs += jvmOptions }

Slide 66

Slide 66

Docker image creation App Platform is read-only: Precompiled templates Shell script creation!

Slide 67

Slide 67

Dockerfile FROM gradle:7.0-jdk16 as build COPY —chown=gradle:gradle . /home/gradle/src WORKDIR /home/gradle/src RUN gradle clean check precompileJte installDist —no-daemon ############### FROM openjdk:16-slim-buster RUN addgroup —system flipbit && adduser —system flipbit —ingroup flipbit USER flipbit:flipbit WORKDIR /flipbit-app COPY —from=build /home/gradle/src/build/ /flipbit-app COPY —from=build /home/gradle/src/jte-classes/ /jte-classes EXPOSE 7000 ENTRYPOINT [ “/flipbit-app/install/flipbit-app/bin/flipbit-app” ]

Slide 68

Slide 68

Dockerfile - agent support // download the lightstep agent as part of our bundle // so we can have autoinstrumentation def agentVersion = “0.16.0” task downloadAgent(type: Download) { mkdir “.gradle/agent/” src “https://github.com/lightstep/otel-launcher-java/releases/download/${agentVersion}/lightstep-opentelemetry-ja vaagent-${agentVersion}.jar” dest new File(“.gradle/agent”, “lightstep-opentelemetry-agent-${agentVersion}.jar”) onlyIfModified true overwrite false } task copyAgent(type: Copy) { dependsOn ‘downloadAgent’ from(“.gradle/agent/lightstep-opentelemetry-agent-${agentVersion}.jar”) { rename “(.*)-${agentVersion}.jar”, ‘$1.jar’ } into “build/install/flipbit-app/agent/” } copyAgent.mustRunAfter installDist Download & copy agent via gradle Add env vars: LS_SERVICE_NAME & FLIPBIT_APP_OPTS

Slide 69

Slide 69

Persistent sessions across restarts this.app = Javalin.create(config -> { … config.sessionHandler(this::sqlSessionHandler); });

Slide 70

Slide 70

Persistent sessions across restarts private SessionHandler sqlSessionHandler() { SessionHandler sessionHandler = new SessionHandler(); SessionCache sessionCache = new DefaultSessionCache(sessionHandler); DatabaseAdaptor databaseAdaptor = new DatabaseAdaptor(); databaseAdaptor.setDatasource(dataSource); JDBCSessionDataStoreFactory jdbcSessionDataStoreFactory = new JDBCSessionDataStoreFactory(); jdbcSessionDataStoreFactory.setDatabaseAdaptor(databaseAdaptor); sessionCache.setSessionDataStore(jdbcSessionDataStoreFactory.getSessionDataStore(sessionHandler)); sessionHandler.setSessionCache(sessionCache); sessionHandler.setHttpOnly(true); } return sessionHandler;

Slide 71

Slide 71

The missing parts So much… all the time Development & Testing Build Metrics Ops Security UI

Slide 72

Slide 72

Missing: Development & Testing Build high level classes around low level DAOs E2E-Testing Don’t run flyway for all unit tests, don’t recreate h2 db for every test class Unique incrementing ids per organization Easy local development: offline support (no oauth) H2/Postgres gap

Slide 73

Slide 73

Build GraalVM all the things - reduces monitoring Reproducible builds

Slide 74

Slide 74

Missing: Metrics Business metrics/health endpoints Analytics (not needed with web logs?) Monitoring/APM

Slide 75

Slide 75

Missing: Ops Terraform setup for DO Rate Limiting (Javalin built-in?) Terraform based monitoring (for example Checkly) APM: Every agent supports Spring Boot and then nothing…

Slide 76

Slide 76

Missing: Security Add all the fancy security headers, CORS X-XXS-Protection , X-Frame-Options , Content-Security-Policy, X-Content-Type-Options . See also OWASP Secure Header Project Add seccomp policy to never fork processes Become smarter about landlock as a java security manager replacement

Slide 77

Slide 77

Missing: UI Rework UI to make it acceptable

Slide 78

Slide 78

Summary Don’t under estimate the work There is no failure! Keep it fun! Combining different libraries means a lot of learning Also, who wants to test? :-) PaaS > K8s

Slide 79

Slide 79

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale

Slide 80

Slide 80

Discussion What technologies would you use? Where did I go wrong?

Slide 81

Slide 81

Thanks for listening Q&A Alexander Reelsen alr@spinscale.de | @spinscale