Importing a Sitemap XML into Google Sheets

Thu, 05 Jan 2023 00:00:00 +0100

The Goal

In the following short article, we want to import data from an existing sitemap XML file into a new Google Sheet document.

The sheet must pull the sitemap via HTTP protocol and extract all the URLs from the sitemap and insert them. into the sheet.

Implementation

Now let’s implement it .. it only takes 1 minute …

For demonstration purpose, I’m going to use the sitemap from my old blog, to be found at https://www.hascode.com/sitemap.xml.

Converting XML Schema (XSD) to Protocol Buffers (Protobuf)

Wed, 02 Feb 2022 00:00:00 +0100

Sometimes one needs to derive a Google Protocol Buffers schema from an XML schema .. e.g. from an Enterprise Architect Export.

Tool used here: schema2proto (GitLab project)

Steps

Download schema2proto-lib from the global Maven repository: https://search.maven.org/search?q=schema2proto
Run Schema2Proto against a give XSD schema file and with a given output directory:

java -jar schema2proto-lib-1.53.jar Schema2Proto --outputDirectory=src/main/protobuf input.xsd

A yaml config file may be given instead of cli parameters:

java -jar schema2proto-lib-1.53.jar Schema2Proto ----configFile=config.yaml input.xsd

Logback and Spring Boot - Change Log Level to custom format

Thu, 19 Aug 2021 00:00:00 +0200

Create a custom converter

This class converts the well known log levels to a custom format

CustomLogLevelConverter.java

package com.hascode;

public class CustomLogLevelConverter extends ClassicConverter {
    @Override
    public String convert(ILoggingEvent event) {
        switch (event.getLevel().toInt()) {
            case Level.ERROR_INT:
                return "ERROR!!!";
            case Level.WARN_INT:
                return "WARN!!";
            case Level.INFO_INT:
                return "INFO!";
            case Level.TRACE_INT:
                return "DEBUG";
            default:
                return event.getLevel().toString();
        }
    }
}

Register the converter

The following Logback config includes some defaults and registers our custom converter.

JUnit5 Java Maven Snippet

Fri, 14 May 2021 00:00:00 +0200

Goals

Use JUnit Maven BOM for version alignment
Add minimal dependencies for JUnit5 Java Projekt

Setup

Excerpt from the Maven project’s pom.xml:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.junit</groupId>
      <artifactId>junit-bom</artifactId>
      <version>5.7.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

JUnit5 does not work with older versions of the Maven Compiler Plugin and the Surefire Plugin used for test execution. Setting their versions in the pom.xml is done like this:

Software Architecture Exploration and Validation with jqAssistant, Neo4j and Cypher

Sun, 31 Dec 2017 00:00:00 +0100

I have written about other software system analyzing and validation tools before but today I would like to introduce a new tool named jqAssistant that supports software architects, developers and analysts in a variety of tasks like analyzing given structures, validating architectural or quality constraints and generating reports.

Therefore jqAssistant analyzes given projects or artifacts and stores the gathered information – that is enriched by a variety of existing plugin-ins – in a Neo4j graph database.

Writing BDD-Style Webservice Tests with Karate and Java

Thu, 06 Apr 2017 00:00:00 +0200

There is a new testing framework out there called Karate that is build on top of the popular Cucumber framework.

Karate makes it easy to script interactions with out web-services under test and verify the results. In addition it offers us a lot of useful features like parallelization, script/feature re-use, data-tables, JsonPath and XPath support, gherkin syntax, switchable staging-configurations and many others.

In the following tutorial we’ll be writing different scenarios and features for a real-world RESTful web-service to demonstrate some of its features.

JAX-RS Server API Snippets

Sun, 28 Sep 2014 00:00:00 +0200

Because a lot of my current projects are using JAX-RS in different versions I’d like to write down and share some frequently used snippets for implementing RESTful web-services with the JAX-RS specification here.

Using RegEx in Path Expressions

Sometimes we need to extract multiple parameters from a path expression e.g. in the following example where year, month and day are fragments if the path.

@GET
@Path("/orders/{year:\\d{4}}-{month:\\d{2}}-{day:\\d{2}}")
@Produces(MediaType.TEXT_PLAIN)
public Response getOrders(@PathParam("year") final int year, @PathParam("month") final int month, @PathParam("day") final int day) {
	return Response.ok("Year: " + year + ", month: " + month + ", day: " + day).build();
}

XMLBeam: Snippets and Examples

Tue, 22 Jul 2014 00:00:00 +0200

XMLBeam is an interesting library using an approach of projecting parts of an XML DOM tree into Java using some simple interfaces, annotations and XPath expressions.

In the following article, I’d like to share three experiments of mine with this library for reading, writing XML and parsing a live RSS feed.

RSS Feed Projection Interface

Dependencies

Using Maven, we need to add only one dependency to our pom.xml:

Creating and Packaging a Game in Java FX 2.2

Sun, 23 Jun 2013 00:00:00 +0200

It’s been a long way for Java FX from the days of the F3 project the current release 2.2. Today there are many options how to create a Java FX application .. you may be using Java, Scala, Groovy or Visage, you may create your application in a programmatic way using the comfortable integrated builders or you may create your views using XML layouts and easy data-bindings with a few annotations.

Creating Rich Clients with Apache Pivot

Sun, 19 May 2013 00:00:00 +0200

Apache Pivot is a modern framework to create rich clients as desktop applications or to run in a web browser.

It offers nice GUI elements, supports XML/WTKX templates, data bindings, JVM scripting languages and much more.

In the following short examples I’m going to create a desktop application to open a file browser, select a file and output the selected file’s name, first using a programmatic approach to create the user interface, and afterwards using XML/WTKX templates.

Creating slim Database Projections using JPA2 Constructor Expressions

Sun, 14 Apr 2013 00:00:00 +0200

One common question that you may encounter one day when using object-relational-mapping in your application is how to slim down data that you’re retrieving from the persistence layer down to a specific subset for your use-case in an efficient manner and without using complex additional mapping frameworks. In some situations you might declare lazy loaded fields but another approach that I’d like to share with you here are JPA2 constructor expressions.

Snippet: XML-Pull-Parser with XPP3

Fri, 28 Dec 2012 00:00:00 +0100

When there is XML to be parsed sometimes we’re chosing a DOM parser, sometimes a SAX parser and sometimes we’re using an XML pull parser, especially on Android.

The following short snippet shows how to use the xpp3 xml pull parse to fetch some information in xml format from a public issue tracker and to extract issue details.

Adding Dependencies

Just one dependency needed here: xpp3

SBT

Simply use the following build.sbt

Contract-First Web-Services using JAX-WS, JAX-B, Maven and Eclipse

Tue, 23 Aug 2011 00:00:00 +0200

Using the contract-first approach to define a web service offers some advantages in contrast to the code-first approach.

In the following tutorial we’re going to take a look at some details of this approach and we’re going to implement a real SOAP service using JAX-WS, Maven and the Eclipse IDE.

Finally we’re going to run our service implementation on an embedded Jetty instance and we’re going to take a look at soapUI and how to test our service using this neat tool.

Google Docs Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Importing XML Data in Google Sheets

Use the XMLIMPORT function

The syntax is XMLIMPORT("url", "xpath")

e.g. for importing an XML sitemap

=IMPORTXML("https://www.hascode.com/sitemap.xml", "//\*[local-name()='url']/*[local-name()='loc']")

Select the preceding cell in Google Sheets

=INDIRECT(ADDRESS(ROW()-1,COLUMN())

Create a Todo List Counter in Google Sheets

Each empty cell is a todo, each filled cell is done.

That allows us the following output, that counts all preceding cells of each type:

="Done: "&(COUNTA(B3:INDIRECT(ADDRESS(ROW()-1,COLUMN())))&" Todo: " &COUNTIF(B3:INDIRECT(ADDRESS(ROW()-1,COLUMN())),""))

This outputs a label like "Done: 35 Todo: 142"

Java Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Remote Debug a Pod’s Java Process

Simple steps for remote debugging a Java process running on a k8 pod:

Edit deployment and add the following parameters to the Java start line: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=127.0.0.1:5005

Also add the following port mapping at the section container → ports in the deployment:
```
- containerPort: 5005
  protocol: TCP
```
Safe, wait for the new pods and then add a port forward for port 5005 for this pod:
```
kubectl port-forward podname 5005
```

Liquibase Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Precondition: Postgres Schema exists

Using this precondition should run a migration only if the schema my_schema exists.

<preConditions onFail="CONTINUE">
    <sqlCheck expectedResult="1">
        SELECT COUNT(1) FROM information_schema.schemata WHERE schema_name = 'my_schema';
    </sqlCheck>
</preConditions>

Resources:

https://docs.liquibase.com/concepts/changelogs/preconditions.html

Spring Boot Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Define and Configure Log Groups

This allows to configure a group of loggers at the same time

Define a log group named myaspect with two packages
application.properties
```
logging.group.myaspect=com.hascode.package1,com.hascode.package2
```
Configure the log group and set all loggers to level TRACE
application.properties
```
logging.level.myaspect=TRACE
```

This is also possible as parameter on startup

java -Dlogging.level.myaspect=TRACE myapp.jar

Use JUnit 5 with Spring Boot

Use newer versions of Surefire and Failsafe plugins:

<properties>
[..]
  <maven-failsafe-plugin.version>2.22.0</maven-failsafe-plugin.version>
  <maven-surefire-plugin.version>2.22.0</maven-surefire-plugin.version>
</properties>

XML Snippets

Mon, 01 Mar 2010 00:00:00 +0100

Ignore Namespaces in XPath Query

e.g. Query for all xxx nodes ignoring their namespace:

xmllint --xpath '//*[local-name()="xxx"]' input.xml

An example parsing URLs from a sitemap XML.

The URLs are located in //url/loc where all nodes are bound to the namespace http://www.sitemaps.org/schemas/sitemap/0.9.

The following query ignores the namespace

xmllint --xpath '//*[local-name()="url"]/*[local-name()="loc"]' sitemap.xml

Pretty Print XML in the Console using xmllint

echo '<blogs><blog url="https://www.hascode.com/">hasCode.com</blog></blogs>' | xmllint --format -
<?xml version="1.0"?>
<blogs>
    <blog url="https://www.hascode.com/">hasCode.com</blog>
</blogs>

Xml on Micha Kops' Tech Notes

Importing a Sitemap XML into Google Sheets

The Goal

Implementation

Converting XML Schema (XSD) to Protocol Buffers (Protobuf)

Steps

Logback and Spring Boot - Change Log Level to custom format

Create a custom converter

Register the converter

JUnit5 Java Maven Snippet

Setup

Software Architecture Exploration and Validation with jqAssistant, Neo4j and Cypher

Writing BDD-Style Webservice Tests with Karate and Java

JAX-RS Server API Snippets

Using RegEx in Path Expressions

XMLBeam: Snippets and Examples

Dependencies

Creating and Packaging a Game in Java FX 2.2

Creating Rich Clients with Apache Pivot

Creating slim Database Projections using JPA2 Constructor Expressions

Snippet: XML-Pull-Parser with XPP3

Adding Dependencies

SBT

Contract-First Web-Services using JAX-WS, JAX-B, Maven and Eclipse

Google Docs Snippets

Importing XML Data in Google Sheets

Select the preceding cell in Google Sheets

Create a Todo List Counter in Google Sheets

Java Snippets

Remote Debug a Pod’s Java Process

Liquibase Snippets

Precondition: Postgres Schema exists

Spring Boot Snippets

Define and Configure Log Groups

Use JUnit 5 with Spring Boot

XML Snippets

Ignore Namespaces in XPath Query

Pretty Print XML in the Console using xmllint