Bitmessage/PyBitmessage-2024-08-21
Archived
2
0
Fork 0
Code Issues 168 Pull Requests 53 Actions Packages Projects Releases 5 Wiki Activity

flatpak-builder support #1618

Closed
zciendor wants to merge 8 commits from v0.6 into v0.6
pull from: v0.6
merge into: Bitmessage:v0.6
Conversation 27 Commits 8 Files Changed 5 +169
zciendor commented 2020-04-30 00:36:46 +02:00 (Migrated from github.com)
Copy Link

Recent distributions like Debian bullseye don't support QT4 anymore. This pull request provides a flatpak-builder configuration and instructions how to package PyBitmessage as a flatpak application which bundles QT4.

This pull request is also a necessary requirement for https://github.com/Bitmessage/PyBitmessage/issues/1607

Recent distributions like `Debian bullseye` don't support QT4 anymore. This pull request provides a `flatpak-builder` configuration and instructions how to package PyBitmessage as a flatpak application which bundles QT4. This pull request is also a necessary requirement for https://github.com/Bitmessage/PyBitmessage/issues/1607
👍 2
PeterSurda commented 2020-04-30 03:08:50 +02:00 (Migrated from github.com)
Copy Link

Thank you, this is great.

Thank you, this is great.
g1itch commented 2020-04-30 10:45:18 +02:00 (Migrated from github.com)
Copy Link

Maybe move the manifest to packages/flatpak?

Maybe move the manifest to packages/flatpak?
👍 1
g1itch (Migrated from github.com) reviewed 2020-04-30 11:33:53 +02:00
INSTALL.md
@ -108,0 +98,4 @@
ebuild.sh - create a package for Gentoo
osx.sh - create a package for OS X
puppy.sh - create a package for Puppy Linux
rpm.sh - create a RPM package
g1itch (Migrated from github.com) commented 2020-04-30 11:33:53 +02:00

It's probably a typo here, should be --install-deps-from=

It's probably a typo here, should be `--install-deps-from=`
zciendor commented 2020-04-30 12:26:41 +02:00 (Migrated from github.com)
Copy Link

Maybe move the manifest to packages/flatpak?

I'll try... need to change some relative paths within the manifest then.

> Maybe move the manifest to packages/flatpak? I'll try... need to change some relative paths within the manifest then.
g1itch commented 2020-04-30 15:26:44 +02:00 (Migrated from github.com)
Copy Link

If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4?
See https://travis-ci.org/github/g1itch/PyBitmessage/jobs/681467891

If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4? See https://travis-ci.org/github/g1itch/PyBitmessage/jobs/681467891
g1itch commented 2020-04-30 16:29:44 +02:00 (Migrated from github.com)
Copy Link

Maybe move the manifest to packages/flatpak?

I'll try... need to change some relative paths within the manifest then.

diff --git a/org.bitmessage.PyBitmessage.json b/org.bitmessage.PyBitmessage.json
index e2bfb73d..c91f5bea 100644
--- a/org.bitmessage.PyBitmessage.json
+++ b/org.bitmessage.PyBitmessage.json
@@ -33,8 +33,8 @@
       ],
       "sources" : [
         {
-          "type" : "dir",
-          "path" : "."
+          "type" : "git",
+          "url" : "https://github.com/Bitmessage/PyBitmessage.git"
         }
       ]
     }
> > Maybe move the manifest to packages/flatpak? > > I'll try... need to change some relative paths within the manifest then. ```patch diff --git a/org.bitmessage.PyBitmessage.json b/org.bitmessage.PyBitmessage.json index e2bfb73d..c91f5bea 100644 --- a/org.bitmessage.PyBitmessage.json +++ b/org.bitmessage.PyBitmessage.json @@ -33,8 +33,8 @@ ], "sources" : [ { - "type" : "dir", - "path" : "." + "type" : "git", + "url" : "https://github.com/Bitmessage/PyBitmessage.git" } ] } ```
zciendor commented 2020-04-30 16:45:18 +02:00 (Migrated from github.com)
Copy Link

If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4?

No, Qt4 is not available in any of the available runtimes on flathub: https://docs.flatpak.org/en/latest/available-runtimes.html

Actually, Qt4 is build within the sandbox and not in Ubuntu.

> If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4? No, Qt4 is not available in any of the available runtimes on flathub: https://docs.flatpak.org/en/latest/available-runtimes.html Actually, Qt4 is build within the sandbox and not in Ubuntu.
zciendor commented 2020-04-30 17:26:56 +02:00 (Migrated from github.com)
Copy Link
-          "type" : "dir",
-          "path" : "."
+          "type" : "git",
+          "url" : "https://github.com/Bitmessage/PyBitmessage.git"

Well, while this works, it would check out the HEAD revision from the remote origin again. This may not be what you want in case you want to build/package the current working copy state.

> ```diff > - "type" : "dir", > - "path" : "." > + "type" : "git", > + "url" : "https://github.com/Bitmessage/PyBitmessage.git" > ``` Well, while this works, it would check out the HEAD revision from the remote origin again. This may not be what you want in case you want to build/package the current working copy state.
zciendor (Migrated from github.com) reviewed 2020-04-30 17:32:12 +02:00
INSTALL.md
@ -108,0 +98,4 @@
ebuild.sh - create a package for Gentoo
osx.sh - create a package for OS X
puppy.sh - create a package for Puppy Linux
rpm.sh - create a RPM package
zciendor (Migrated from github.com) commented 2020-04-30 17:32:12 +02:00

It's probably a typo here, should be --install-deps-from=

Fixed. The updated build instructions are now in packages/flatpak/README.md.

> It's probably a typo here, should be `--install-deps-from=` Fixed. The updated build instructions are now in [packages/flatpak/README.md](https://github.com/Bitmessage/PyBitmessage/pull/1618/files#diff-8421532ac371e215f58a7bbb8a782c8a).
zciendor commented 2020-04-30 17:38:38 +02:00 (Migrated from github.com)
Copy Link

If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4?
See https://travis-ci.org/github/g1itch/PyBitmessage/jobs/681467891

Timeout (20 minutes) reached.

flatpak-builder has a cache directory. Qt4 won't be rebuild if this exists from a previous build and the inputs for a build step haven't changed. Not sure if this could be configured in travis-ci somehow...

> If we already have all dependencies installed on Ubuntu bionic, is it possible to skip building Qt4? > See https://travis-ci.org/github/g1itch/PyBitmessage/jobs/681467891 ``` Timeout (20 minutes) reached. ``` flatpak-builder has a cache directory. Qt4 won't be rebuild if this exists from a previous build and the inputs for a build step haven't changed. Not sure if this could be configured in travis-ci somehow...
g1itch commented 2021-02-04 18:08:45 +01:00 (Migrated from github.com)
Copy Link

Maybe you can take my qt5-wip branch and try to build with Qt5?

Maybe you can take my [qt5-wip branch](https://github.com/g1itch/PyBitmessage/tree/qt5-wip) and try to build with Qt5?
PeterSurda (Migrated from github.com) requested changes 2021-03-17 12:55:20 +01:00
PeterSurda (Migrated from github.com) left a comment

I tested it and it works fine, now I can use PyBM on Ubuntu 20.04.

For integration into the pipeline I'll have to cache qt and pyqt as they take long to build, but I'll deal with that. Bump the sip version and update docs as I requested.

I tested it and it works fine, now I can use PyBM on Ubuntu 20.04. For integration into the pipeline I'll have to cache qt and pyqt as they take long to build, but I'll deal with that. Bump the `sip` version and update docs as I requested.
packages/flatpak/README.md
@ -0,0 +15,4 @@
git clone git://github.com/Bitmessage/PyBitmessage.git
cd PyBitmessage/
git submodule update --init --recursive
flatpak-builder --install --user -install-deps-from=flathub --force-clean --state-dir=build/.flatpak-builder build/_flatpak org.bitmessage.PyBitmessage.json
PeterSurda (Migrated from github.com) commented 2021-03-17 12:43:23 +01:00

flatpak-builder --install --user -install-deps-from=flathub --force-clean --state-dir=build/.flatpak-builder build/_flatpak packages/flatplak/org.bitmessage.PyBitmessage.json

`flatpak-builder --install --user -install-deps-from=flathub --force-clean --state-dir=build/.flatpak-builder build/_flatpak packages/flatplak/org.bitmessage.PyBitmessage.json`
packages/flatpak/org.bitmessage.PyBitmessage.json
@ -0,0 +18,4 @@
"sources": [
{
"type": "archive",
"url": "https://www.riverbankcomputing.com/static/Downloads/sip/4.19.22/sip-4.19.22.tar.gz",
PeterSurda (Migrated from github.com) commented 2021-03-17 12:53:36 +01:00

"url": "https://www.riverbankcomputing.com/static/Downloads/sip/4.19.25/sip-4.19.25.tar.gz",

` "url": "https://www.riverbankcomputing.com/static/Downloads/sip/4.19.25/sip-4.19.25.tar.gz",`
packages/flatpak/org.bitmessage.PyBitmessage.json
@ -0,0 +19,4 @@
{
"type": "archive",
"url": "https://www.riverbankcomputing.com/static/Downloads/sip/4.19.22/sip-4.19.22.tar.gz",
"sha256": "e1b768824ec1a2ee38dd536b6b6b3d06de27b00a2f5f55470d1b512306e3be45"
PeterSurda (Migrated from github.com) commented 2021-03-17 12:53:54 +01:00

"sha256": "b39d93e937647807bac23579edbff25fe46d16213f708370072574ab1f1b4211"

` "sha256": "b39d93e937647807bac23579edbff25fe46d16213f708370072574ab1f1b4211"`
PeterSurda commented 2021-03-17 12:55:40 +01:00 (Migrated from github.com)
Copy Link

Also rebase.

Also rebase.
PeterSurda commented 2021-03-18 10:45:53 +01:00 (Migrated from github.com)
Copy Link

It would be great to be able to build the modules separately and upload them to a repo. I don't have time to find out how to do this myself. Can someone do it? Once I have the ability build and publish individual modules, I can make it into an incremental buildbot job.

It would be great to be able to build the modules separately and upload them to a repo. I don't have time to find out how to do this myself. Can someone do it? Once I have the ability build and publish individual modules, I can make it into an incremental buildbot job.
zciendor commented 2021-03-24 17:27:40 +01:00 (Migrated from github.com)
Copy Link

Once I have the ability build and publish individual modules, I can make it into an incremental buildbot job.

What do you mean exactly by "publishing" individual modules? If you mean publishing as individual flatpaks I don't think this is possible as flatpaks can only declare dependencies on "runtimes". So you would have to publish and maintain your own "runtime" on flathub, seems a bit overkill to me.

But if you just want to break up the build process that could probably be achieved by splitting the manifest into two or three, four, ... files. You would just have to add simple copy commands to backup the build artifacts to another location outside the build dir and copy them back into the build dir in the final manifest where you build PyBitmessage.

But since this depends highly on your build environment I think you have to do the separation on your own. The --stop-at parameter to flatpak-builder might help you. You can take the current manifest and stop at the last dependency and inspect the build directory:

flatpak-builder --user --install-deps-from=flathub --state-dir=build/.flatpak-builder --build-only --stop-at=python-sip build/_flatpak packages/flatpak/org.bitmessage.PyBitmessage.json.

The only issue is that you can't simply "continue" after using the --stop-at option as flatpak-builder always wants a clean/empty build directory and aborts if that is not the case.

The solution could be do use different build dirs for each step (per manifest) and add a copy command(s) as the first thing to do in the final PyBitmessage module / manifest.

> Once I have the ability build and publish individual modules, I can make it into an incremental buildbot job. What do you mean exactly by "publishing" individual modules? If you mean publishing as individual flatpaks I don't think this is possible as flatpaks can only declare dependencies on "runtimes". So you would have to publish and maintain your own "runtime" on flathub, seems a bit overkill to me. But if you just want to break up the build process that could probably be achieved by splitting the manifest into two or three, four, ... files. You would just have to add simple copy commands to backup the build artifacts to another location outside the build dir and copy them back into the build dir in the final manifest where you build PyBitmessage. But since this depends highly on your build environment I think you have to do the separation on your own. The `--stop-at` parameter to `flatpak-builder` might help you. You can take the current manifest and stop at the last dependency and inspect the build directory: `flatpak-builder --user --install-deps-from=flathub --state-dir=build/.flatpak-builder --build-only --stop-at=python-sip build/_flatpak packages/flatpak/org.bitmessage.PyBitmessage.json`. The only issue is that you can't simply "continue" after using the `--stop-at` option as `flatpak-builder` always wants a clean/empty build directory and aborts if that is not the case. The solution could be do use different build dirs for each step (per manifest) and add a copy command(s) as the first thing to do in the final `PyBitmessage module` / manifest.
PeterSurda commented 2021-03-24 18:54:45 +01:00 (Migrated from github.com)
Copy Link

Well, I'm not really familiar with flatpak, but I can compare it to what I do with brew for mac build. There I also have to build some dependencies, such as qt and sip, as the binaries aren't always publicly available. What I do is that the build first tries to install the dependency from a URL (private 'repo'), and if it fails, builds it and uploads it to said repo. Next time the build runs and the dependency hasn't changed (no version bump), no need to build it as it's installed from the URL. I was thinking I can do the same with flatpak but it isn't clear to me how to create my own repo. With brew, I can just specify a URL to install from, so any web server will do.

Each build is started from a clean environment.

Well, I'm not really familiar with flatpak, but I can compare it to what I do with brew for mac build. There I also have to build some dependencies, such as qt and sip, as the binaries aren't always publicly available. What I do is that the build first tries to install the dependency from a URL (private 'repo'), and if it fails, builds it and uploads it to said repo. Next time the build runs and the dependency hasn't changed (no version bump), no need to build it as it's installed from the URL. I was thinking I can do the same with flatpak but it isn't clear to me how to create my own repo. With brew, I can just specify a URL to install from, so any web server will do. Each build is started from a clean environment.
zciendor commented 2021-03-26 10:55:28 +01:00 (Migrated from github.com)
Copy Link

What about simply persisting the whole state-dir (see --state-dir=build/.flatpak-builder) across build jobs? Everything is in there. Python and QT4 won't be rebuild if their metadata JSONs didn't change:

$ flatpak-builder --user --install-deps-from=flathub --state-dir=build/.flatpak-builder --build-only --stop-at=python-sip build/_flatpak packages/flatpak/org.bitmessage.PyBitmessage.json
Dependency Sdk: org.freedesktop.Sdk 18.08
Updating org.freedesktop.Sdk/x86_64/18.08
Nothing to do.
Dependency Runtime: org.freedesktop.Platform 18.08
Updating org.freedesktop.Platform/x86_64/18.08
Nothing to do.
Downloading sources
Stopping at module python-sip
Starting build of org.bitmessage.PyBitmessage
Cache hit for python-2.7, skipping build
Cache hit for qt4, skipping build
Stopping at module python-sip
Everything cached, checking out from cache
Pruning cache
What about simply persisting the whole `state-dir` (see `--state-dir=build/.flatpak-builder`) across build jobs? Everything is in there. Python and QT4 won't be rebuild if their metadata JSONs didn't change: ``` $ flatpak-builder --user --install-deps-from=flathub --state-dir=build/.flatpak-builder --build-only --stop-at=python-sip build/_flatpak packages/flatpak/org.bitmessage.PyBitmessage.json Dependency Sdk: org.freedesktop.Sdk 18.08 Updating org.freedesktop.Sdk/x86_64/18.08 Nothing to do. Dependency Runtime: org.freedesktop.Platform 18.08 Updating org.freedesktop.Platform/x86_64/18.08 Nothing to do. Downloading sources Stopping at module python-sip Starting build of org.bitmessage.PyBitmessage Cache hit for python-2.7, skipping build Cache hit for qt4, skipping build Stopping at module python-sip Everything cached, checking out from cache Pruning cache ```
PeterSurda commented 2021-03-26 11:02:48 +01:00 (Migrated from github.com)
Copy Link

What about simply persisting the whole state-dir (see --state-dir=build/.flatpak-builder) across build jobs?

This doesn't fit into the design of my setup. All build VMs and containers are ephemeral.

> What about simply persisting the whole `state-dir` (see `--state-dir=build/.flatpak-builder`) across build jobs? This doesn't fit into the design of my setup. All build VMs and containers are ephemeral.
zciendor commented 2021-03-26 11:05:07 +01:00 (Migrated from github.com)
Copy Link

Note that the state-dir is not the the build dir. build dir must be clean and empty before every build. But the purpose of the state-dir is to be persisted across builds to achieve all the caching and speedups you are looking for.

Note that the `state-dir` is not the the `build dir`. `build dir` must be clean and empty before every build. But the purpose of the `state-dir` is to be persisted across builds to achieve all the caching and speedups you are looking for.
zciendor commented 2021-03-26 11:07:01 +01:00 (Migrated from github.com)
Copy Link

Can't you rsync or archive the whole state-dir at the end of a build job and upload it to your server/repository and sync it back to the VM at the beginning of a build job?

Can't you rsync or archive the whole `state-dir` at the end of a build job and upload it to your server/repository and sync it back to the VM at the beginning of a build job?
PeterSurda commented 2021-03-26 11:08:50 +01:00 (Migrated from github.com)
Copy Link

That's basically a kludge version of what I want.

That's basically a kludge version of what I want.
zciendor commented 2021-03-26 11:13:51 +01:00 (Migrated from github.com)
Copy Link

This doesn't fit into the design of my setup. All build VMs and containers are ephemeral.

But usually you can mount some specific persistent volumes/folders/dirs to a ephermeral VM.

The .flatepak-builder directory is 5.5G right now. You would save a lot of build job time and network bandwidth if you just reserve some space on the build machine to be mounted persistent across builds.

> This doesn't fit into the design of my setup. All build VMs and containers are ephemeral. But usually you can mount some specific persistent volumes/folders/dirs to a ephermeral VM. The `.flatepak-builder` directory is 5.5G right now. You would save a lot of build job time and network bandwidth if you just reserve some space on the build machine to be mounted persistent across builds.
PeterSurda commented 2021-03-26 11:38:50 +01:00 (Migrated from github.com)
Copy Link

Technically that may work, but isn't suitable for easy deployment and maintenance, it creates another item in infrastructure I have to worry about. Compared that to what I do with brew, it just re-uses an existing web server and cache. How about this: https://pagure.io/flatpak-module-tools , doesn't it do what I want?

Technically that may work, but isn't suitable for easy deployment and maintenance, it creates another item in infrastructure I have to worry about. Compared that to what I do with brew, it just re-uses an existing web server and cache. How about this: https://pagure.io/flatpak-module-tools , doesn't it do what I want?
zciendor commented 2021-03-26 13:05:33 +01:00 (Migrated from github.com)
Copy Link

How about this: https://pagure.io/flatpak-module-tools , doesn't it do what I want?

As far as I understand this has something to do with "Fedora modules"... not "flatpak modules". It's just using flatpak to build Fedora modules.

The notion of a "module" within the flatpak manifest is nothing which can be exported and deployed as a standalone artifact. Flatpak only has the notion of standalone and sandboxed "applications" which have a single dependency on a runtime. So unless you want to become a flatpak "runtime developer" and "runtime maintainer", an "application" is the only thing you can produce. There are no "modules" from a build artifact / repository / deployment perspective. "Modules" only exist from a source code or "build steps" perspective. The source code can be structured into "modules". But "build artifacts" that can be exported are only standalone "applications".

> How about this: https://pagure.io/flatpak-module-tools , doesn't it do what I want? As far as I understand this has something to do with "Fedora modules"... not "flatpak modules". It's just using flatpak to build Fedora modules. The notion of a "module" within the flatpak manifest is nothing which can be exported and deployed as a standalone artifact. Flatpak only has the notion of standalone **and sandboxed** "applications" which have a single dependency on a runtime. So unless you want to become a flatpak "runtime developer" and "runtime maintainer", an "application" is the only thing you can produce. There are no "modules" from a build artifact / repository / deployment perspective. "Modules" only exist from a source code or "build steps" perspective. The source code can be structured into "modules". But "build artifacts" that can be exported are only standalone "applications".
PeterSurda commented 2021-03-26 16:01:54 +01:00 (Migrated from github.com)
Copy Link

Can I do something like a wget && tar xJf module.tar.xz with fallback to flatpak build-module module1 && tar cJf module1.tar.xz && upload?

Can I do something like a `wget && tar xJf module.tar.xz` with fallback to `flatpak build-module module1 && tar cJf module1.tar.xz && upload`?
zciendor commented 2021-03-26 21:39:44 +01:00 (Migrated from github.com)
Copy Link

Sorry, I'm only familiar with flatpak-builder (https://docs.flatpak.org/en/latest/building-introduction.html#flatpak-builder).

Sorry, I'm only familiar with `flatpak-builder` (https://docs.flatpak.org/en/latest/building-introduction.html#flatpak-builder).
PeterSurda commented 2021-03-29 07:59:58 +02:00 (Migrated from github.com)
Copy Link

Ok, how about you just fix the other things I requested (documentation, bump sip, rebase), and the buildbot job will be run manually for now, the splitting into steps I'll deal with separately. This way at least I can have an official flatpak that can be distributed.

Ok, how about you just fix the other things I requested (documentation, bump sip, rebase), and the buildbot job will be run manually for now, the splitting into steps I'll deal with separately. This way at least I can have an official flatpak that can be distributed.
g1itch commented 2021-05-04 18:34:52 +02:00 (Migrated from github.com)
Copy Link

Maybe you can take my qt5-wip branch and try to build with Qt5?

Oh, python2 now also needs that git submodules. Too bad.

> Maybe you can take my [qt5-wip branch](https://github.com/g1itch/PyBitmessage/tree/qt5-wip) and try to build with Qt5? Oh, python2 now also needs that git submodules. Too bad.
g1itch commented 2021-05-04 18:36:31 +02:00 (Migrated from github.com)
Copy Link

I guess now this should be considered abandoned. Duplicates: #1755.

I guess now this should be considered abandoned. Duplicates: #1755.
PeterSurda commented 2021-05-04 18:38:15 +02:00 (Migrated from github.com)
Copy Link

@g1itch yes it's a duplicate, the new one has split dependencies and app, so that you don't have to build QT4 everytime you are building a new flatpak of the app.

@g1itch yes it's a duplicate, the new one has split dependencies and app, so that you don't have to build QT4 everytime you are building a new flatpak of the app.
This repo is archived. You cannot comment on pull requests.
Reviewers
No reviewers
PeterSurda
Labels
Clear labels   
bug

Something isn't working as it's supposed to

  
build

Related to building or build system

  
dependencies

Pull requests that update a dependency file

  
developers

The issue is relevant mainly for developers rather than users

  
documentation

   
duplicate

   
enhancement

New feature

  
formatting

Code or UI text formatting

  
invalid

   
legal

   
mobile

Frontend for mobile devices (kivy)

  
obsolete

   
packaging

   
performance

   
protocol

Protocol change

  
question

   
refactoring

   
regression

Some issue, fixed in the past, appears again

  
security

   
test

   
translation

   
usability

   
wontfix
No Label
bug
build
dependencies
developers
documentation
duplicate
enhancement
formatting
invalid
legal
mobile
obsolete
packaging
performance
protocol
question
refactoring
regression
security
test
translation
usability
wontfix
Milestone
Clear milestone
No items
No Milestone
v0.6.4
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Bitmessage/PyBitmessage-2024-08-21#1618
No description provided.
Delete Branch "v0.6"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?