Bitmessage/buildbot_multibuild
6
0
Fork 5
Code Issues 4 Pull Requests Projects Releases Wiki Activity

worker and master splited #9

Merged
PeterSurda merged 1 commits from cis-muzahid/buildbot_multibuild:kivy-test into master 2022-03-09 13:34:02 +01:00
Conversation 1 Commits 1 Files Changed 2 +147 -134
2 changed files with 147 additions and 134 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

125
lib/worker_multibuild.py Normal file
View File

@ -0,0 +1,125 @@
from os import listdir, walk
from os.path import exists, isfile, join, islink, isdir
import requests
import re
request_data = {
"project": "testproject",
"comments": "testcomment",
}
request_headers = {
"Content-Type": "application/x-www-form-urlencoded",
"Accept": "text/plain",
}
ty = "/change_hook/base"
path =".buildbot"
dockerfile_extra_contents = {}
dockerfile_extra_contents['focal'] = """
# Buildbot
RUN apt-get install -yq --no-install-suggests --no-install-recommends \
buildbot-worker git subversion python3-dev libffi-dev python3-setuptools \
python3-pip dumb-init curl openssh-client wget
# buildbot entrypoint
RUN wget -O /usr/local/bin/buildbot_entrypoint.sh https://git.bitmessage.org/Bitmessage/buildbot-scripts/raw/branch/master/docker/bionic/entrypoint.sh
RUN chmod +x /usr/local/bin/buildbot_entrypoint.sh
RUN echo 'buildbot ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER buildbot
ENTRYPOINT /usr/local/bin/buildbot_entrypoint.sh "$BUILDMASTER" "$WORKERNAME" "$WORKERPASS"
"""
dockerfile_extra_contents['bionic'] = """
# Buildbot
RUN apt-get install -yq --no-install-suggests --no-install-recommends \
buildbot-slave git subversion python3-dev libffi-dev python3-setuptools \
python3-pip dumb-init curl openssh-client wget
# buildbot entrypoint
RUN wget -O /usr/local/bin/buildbot_entrypoint.sh https://git.bitmessage.org/Bitmessage/buildbot-scripts/raw/branch/master/docker/bionic/entrypoint.sh
RUN chmod +x /usr/local/bin/buildbot_entrypoint.sh
RUN echo 'buildbot ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER buildbot
ENTRYPOINT /usr/local/bin/buildbot_entrypoint.sh "$BUILDMASTER" "$WORKERNAME" "$WORKERPASS"
"""
def list_jobs(directory=".buildbot"):
"""
list jobs found in a directory
"""
results = []
files = ["Dockerfile", "build.sh", "test.sh"]
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 12:08:54 +01:00
Outdated
Review
Copy Link

I wouldn't use walk, we only need one level. Also, we need to add a check that Dockerfile, build.sh and test.sh are regular files and not symlinks (probably two checks needed per file so another function should be written for that)

I wouldn't use walk, we only need one level. Also, we need to add a check that `Dockerfile`, `build.sh` and `test.sh` are regular files and not symlinks (probably two checks needed per file so another function should be written for that)
for item in listdir(directory):
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-02 07:17:51 +01:00
Outdated
Review
Copy Link

I think I didn't explain it well. It should look for subdirectories, but only at one level, and should ignore symlinked directories

I think I didn't explain it well. It should look for subdirectories, but only at one level, and should ignore symlinked directories
cis-muzahid commented 2022-03-03 16:27:49 +01:00
Outdated
Review
Copy Link

results = []
files = ["Dockerfile", "build.sh", "test.sh"]
for item in listdir(directory):
print(item)
if isdir(join(directory, item)) and not islink(join(directory, item)):
for file in listdir(join(directory, item)):
if file in files and isfile(join(directory, item, file)) and not islink(join(directory, item, file)):
results.append(file)
return results

Is this logic fine? I haven't remove file name from files while it found in path

results = [] files = ["Dockerfile", "build.sh", "test.sh"] for item in listdir(directory): print(item) if isdir(join(directory, item)) and not islink(join(directory, item)): for file in listdir(join(directory, item)): if file in files and isfile(join(directory, item, file)) and not islink(join(directory, item, file)): results.append(file) return results ```Is this logic fine? I haven't remove file name from files while it found in path```
flag = False
for file in files:
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-04 07:38:25 +01:00
Outdated
Review
Copy Link
for file in files:
    filepath = join(directory, item, file)
    if exists and bad:
       break 2ND LEVEL `for` loop
# here the directory passed check 1
if exists(Dockerfile) and (exists(build.sh) or exists(test.sh)):
    add `item` (DIRECTORY, NOT FILE) to return value
``` for file in files: filepath = join(directory, item, file) if exists and bad: break 2ND LEVEL `for` loop # here the directory passed check 1 if exists(Dockerfile) and (exists(build.sh) or exists(test.sh)): add `item` (DIRECTORY, NOT FILE) to return value ```
cis-muzahid commented 2022-03-07 12:37:05 +01:00
Outdated
Review
Copy Link

With the following condition
if exists(Dockerfile) and (exists(build.sh) or exists(test.sh)):
If we have Dockerfile as a normal file and builds.sh as symlink so its skips Dockerfile as it does not include the directory.
So its clear we have to keep both Dockerfile and builds.sh at same placein the directory

With the following condition ```if exists(Dockerfile) and (exists(build.sh) or exists(test.sh)):``` If we have ```Dockerfile``` as a normal file and ```builds.sh``` as symlink so its skips ```Dockerfile``` as it does not include the directory. So its clear we have to keep both ```Dockerfile and builds.sh ``` at same placein the directory
PeterSurda commented 2022-03-07 13:00:34 +01:00
Outdated
Review
Copy Link

These files are all per directory. And only if all the files in the directory pass checks, will the directory be permitted as a job.

These files are all per directory. And only if all the files in the directory pass checks, will the directory be permitted as a job.
filepath = join(directory, item, file)
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-02 07:18:20 +01:00
Outdated
Review
Copy Link

this is incomplete (doesn't filter directories, sockets, devices, etc). Better would be isfile and not islink

this is incomplete (doesn't filter directories, sockets, devices, etc). Better would be `isfile and not islink`
👍 1
PeterSurda commented 2022-03-07 08:00:43 +01:00
Outdated
Review
Copy Link
if islink(filepath) or not isfile(filepath):
``` if islink(filepath) or not isfile(filepath): ```
👍 1
if islink(filepath) or not isfile(filepath):
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-04 07:38:59 +01:00
Outdated
Review
Copy Link

we should add item, not file to the return array (should be fixed as described in the pseudocode above)

we should add `item`, not `file` to the return array (should be fixed as described in the pseudocode above)
flag = True
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-07 08:01:19 +01:00
Outdated
Review
Copy Link

indent one level up

indent one level up
👍 1
PeterSurda commented 2022-03-07 08:26:47 +01:00
Outdated
Review
Copy Link

also first or shoud be and and missing brackets

also first `or` shoud be `and` and missing brackets
👍 1
break
if flag:
continue
if (exists(join(directory, item, 'Dockerfile')) and exists(join(directory, item, 'build.sh'))) or exists(join(directory, item, 'test.sh')):
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 08:24:54 +01:00
Outdated
Review
Copy Link

remove os_codename, make it a variable, not an argument

remove `os_codename`, make it a variable, not an argument
👍 1
results.append(item)
return results
def _get_dockerfile_contents(jobname):
"""
Read contents of a Dockerfile and add extra contents for the given os_codename
"""
os_codename='bionic'
res = ""
with open(join(path + jobname), "r") as file:
contents = file.read()
# accept any line containing FROM and RUN keywords
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-07 08:07:51 +01:00
Outdated
Review
Copy Link

ENV also is allowed

`ENV` also is allowed
👍 1
res = ""
inside_allowed_command = False
for line in contents:
if re.match(r"(?m)^(FROM|RUN|ENV).*$", line):
inside_allowed_command = True
if inside_allowed_command:
res += line
l = line.strip()
if not l.endswith("\\"):
inside_allowed_command = False
return res + dockerfile_extra_contents[os_codename]
def trigger_child_hooks(buildbotUrl: str, repository, branch, directory=".buildbot"):
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-08 07:55:53 +01:00
Outdated
Review
Copy Link

no jobname
no is_build_script_available
no is_test_script_available

These are all per-jobs variables, extracted from the filesystem in each loop.

no `jobname` no `is_build_script_available` no `is_test_script_available` These are all per-jobs variables, extracted from the filesystem in each loop.
request_url = buildbotUrl + ty
# List all jobs in the directory
jobs = list_jobs(directory)
# Check if build.sh or test.sh exists in each of the jobs
for job in jobs:
build_script_exists = False
test_script_exists = False
if exists(join(directory, job, "build.sh")):
build_script_exists = True
if exists(join(directory, job, "test.sh")):
test_script_exists = True
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 08:23:55 +01:00
Outdated
Review
Copy Link

remove os_codename from arguments

remove `os_codename` from arguments
👍 1
# make a post request
request_data["properties"] = {
"dockerfile": _get_dockerfile_contents(
join(directory, job, "Dockerfile")
),
"build_script_available": is_build_script_available(build_script_exists),
"test_script_available": is_test_script_available(test_script_exists),
"repository": repository,
"branch": branch,
"jobname": job,
}
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-08 07:54:53 +01:00
Outdated
Review
Copy Link

"jobname": job,

` "jobname": job, `
requests.post(request_url, headers=request_headers, data=request_data)

156
multibuild.py
View File

@ -10,7 +10,7 @@ Requires docker
# TODO: write hook job, maybe also a dockerfile?
# TODO: what to do about non-docker jobs
from os import listdir, walk
from os import listdir, walk, getenv
from os.path import exists, isfile, join
import requests
import re
@ -18,70 +18,6 @@ from buildbot.plugins import steps, util
from .lib.renderers import *
ty = "/change_hook/base"
request_headers = {
"Content-Type": "application/x-www-form-urlencoded",
"Accept": "text/plain",
}
request_data = {
"project": "testproject",
"comments": "testcomment",
}
dockerfile_extra_contents_focal = """
# Buildbot
RUN apt-get install -yq --no-install-suggests --no-install-recommends \
buildbot-worker git subversion python3-dev libffi-dev python3-setuptools \
python3-pip dumb-init curl openssh-client wget
# buildbot entrypoint
RUN wget -O /usr/local/bin/buildbot_entrypoint.sh https://git.bitmessage.org/Bitmessage/buildbot-scripts/raw/branch/master/docker/bionic/entrypoint.sh
RUN chmod +x /usr/local/bin/buildbot_entrypoint.sh
RUN echo 'buildbot ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER buildbot
ENTRYPOINT /usr/local/bin/buildbot_entrypoint.sh "$BUILDMASTER" "$WORKERNAME" "$WORKERPASS"
"""
dockerfile_extra_contents_bionic = """
# Buildbot
RUN apt-get install -yq --no-install-suggests --no-install-recommends \
buildbot-slave git subversion python3-dev libffi-dev python3-setuptools \
python3-pip dumb-init curl openssh-client wget
# buildbot entrypoint
RUN wget -O /usr/local/bin/buildbot_entrypoint.sh https://git.bitmessage.org/Bitmessage/buildbot-scripts/raw/branch/master/docker/bionic/entrypoint.sh
RUN chmod +x /usr/local/bin/buildbot_entrypoint.sh
RUN echo 'buildbot ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
USER buildbot
ENTRYPOINT /usr/local/bin/buildbot_entrypoint.sh "$BUILDMASTER" "$WORKERNAME" "$WORKERPASS"
"""
path =".buildbot"
def list_jobs(directory=".buildbot"):
"""
list jobs found in a directory
"""
results = []
for _ in next(walk(directory))[1]:
if exists(join(directory, _, "Dockerfile")) and (
exists(join(directory, _, "build.sh"))
or exists(join(directory, _, "test.sh"))
):
results.append(_)
return results
def find_artifacts(directory="out"):
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-09 07:42:27 +01:00
Outdated
Review
Copy Link

can be removed

can be removed
👍 1
"""
@ -93,96 +29,49 @@ def find_artifacts(directory="out"):
return join(directory, _)
def get_dockerfile_contents(props):
"""
Read contents of a Dockerfile and add extra contents for the given os_codename
"""
with open(join(path + props.getProperty('jobname', default=None)), "r") as file:
contents = file.read()
# remove any line containing CMD or ENTRYFILE keywords
re.sub(r"(?m)^(CMD|ENTRYFILE).*$", "", contents)
return contents + {
"focal": dockerfile_extra_contents_focal,
"bionic": dockerfile_extra_contents_bionic,
}[util.Interpolate("%(prop:os_codename)s")]
def trigger_child_hooks(buildbotUrl: str, os_codename: str, repository, branch, jobname, directory=".buildbot"):
request_url = buildbotUrl + ty
# List all jobs in the directory
jobs = list_jobs(directory)
# Check if build.sh or test.sh exists in each of the jobs
for job in jobs:
build_script_exists = False
test_script_exists = False
if exists(join(directory, job, "build.sh")):
build_script_exists = True
if exists(join(directory, job, "test.sh")):
test_script_exists = True
# make a post request
request_data["properties"] = {
"dockerfile": get_dockerfile_contents(
join(directory, job, "Dockerfile"), os_codename
),
"build_script_available": is_build_script_available(build_script_exists),
"test_script_available": is_test_script_available(test_script_exists),
"repository": repository,
"branch": branch,
"jobname": jobname,
}
requests.post(request_url, headers=request_headers, data=request_data)
def add_parent_step(build_factory):
"""
Add a step to the parent build factory that will trigger the child hooks
"""
build_factory.addStep(
steps.SetPropertyFromCommand(
name="Get OS codename",
command="grep ^VERSION_CODENAME= /etc/os-release | cut -d= -f2",
property="os_codename",
)
)
build_factory.addStep(steps.ShellCommand(
name="create directory",
command=["mkdir", "-p", join(getenv['HOME'], '.local/bin') ]
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 08:24:28 +01:00
Outdated
Review
Copy Link

add mkdir -p

add `mkdir -p`
👍 1
PeterSurda commented 2022-03-01 12:07:07 +01:00
Outdated
Review
Copy Link

should be two separate commands.

should be two separate commands.
))
build_factory.addStep(steps.ShellCommand(
name="download worker",
command=["wget", "-O", "https://git.bitmessage.org/Bitmessage/buildbot_multibuild/raw/branch/master/lib/worker_multibuild.py", join(getenv['HOME'], '.local/bin/worker_multibuild.py')]
))
build_factory.addStep(
steps.ShellCommand(
name="Execute multibuild script",
name="Execute worker script",
command=[
"python",
"multibuild.py",
util.Interpolate("%(prop:jobname)s"),
"python3",
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-02 07:20:58 +01:00
Outdated
Review
Copy Link

python3

`python3`
join(getenv['HOME'], '.local/bin/worker_multibuild.py'),
util.Property('repository'),
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-07 08:10:04 +01:00
Outdated
Review
Copy Link

no jobname in parent. Parent inserts the jobnames into buildbot, doesn't read them from buildbot. Child contains a jobname as a property then.

no `jobname` in parent. Parent inserts the jobnames into buildbot, doesn't read them from buildbot. Child contains a jobname as a property then.
👍 1
PeterSurda commented 2022-03-07 08:10:49 +01:00
Outdated
Review
Copy Link

same for os_codename. Parent inserts os_codename into buildbot.

same for `os_codename`. Parent inserts os_codename into buildbot.
PeterSurda commented 2022-03-07 08:24:37 +01:00
Outdated
Review
Copy Link

perhaps insteaad of URL, but it may not be possible at least without workarounds

getURLForBuild(master, builderid, build_number)

maybe also

util.Property('url')
perhaps insteaad of URL, but it may not be possible at least without workarounds ``` getURLForBuild(master, builderid, build_number) ``` maybe also ``` util.Property('url') ```
👍 1
util.Property('branch'),
"https://buildbot.bitmessage.org",
util.Interpolate("%(prop:os_codename)s"),
util.getURLForBuild(util.Property("url"), util.Property("builderid"), util.Property("buildnumber")),
],
)
)
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-08 07:53:42 +01:00
Outdated
Review
Copy Link

try:
util.Property('url')

try: ` util.Property('url') `
👍 1
def add_child_build_sh_step(build_factory, directory=".buildbot"):
def add_child_sh_steps(build_factory, directory=".buildbot"):
"""
Add a step to the build factory
Add a step to the download, build and test factory
"""
build_factory.addStep(
steps.ShellCommand(
name=util.Interpolate("build_%(prop:jobname)s"),
command=util.Interpolate("%(kw:directory)s/%(prop:jobname)s/test.sh", directory=directory),
command=util.Interpolate("%(kw:directory)s/%(prop:jobname)s/build.sh", directory=directory),
doStepIf=is_build_script_available,
hideStepIf=isnt_build_script_available,
)
)
def add_child_test_sh_step(build_factory, directory=".buildbot"):
"""
Add a step to the build factory
"""
build_factory.addStep(
steps.ShellCommand(
name= util.Interpolate("test_%(prop:jobname)s"),
@ -194,7 +83,7 @@ def add_child_test_sh_step(build_factory, directory=".buildbot"):
if __name__ == "__main__":
# expect jobname, repository, branch, buildbotUrl, os_codename from command line args
# expect jobname, repository, branch, buildbotUrl from command line
import sys
if len(sys.argv) == 6:
@ -202,10 +91,9 @@ if __name__ == "__main__":
repository = sys.argv[2]
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 08:24:08 +01:00
Outdated
Review
Copy Link

no os_codename

no `os_codename`
👍 1
branch = sys.argv[3]
buildbotUrl = sys.argv[4]
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-01 08:24:16 +01:00
Outdated
Review
Copy Link

also here

also here
👍 1
os_codename = sys.argv[5]
trigger_child_hooks(buildbotUrl, os_codename, repository, branch, jobname)
trigger_child_hooks(buildbotUrl, repository, branch)
else:
PeterSurda marked this conversation as resolved
PeterSurda commented 2022-03-09 07:43:51 +01:00
Review
Copy Link

remove buildbotUrl

remove `buildbotUrl`
cis-muzahid commented 2022-03-09 09:36:49 +01:00
Review
Copy Link

Sorry, I think buildbotUrl need to be here

Sorry, I think `buildbotUrl` need to be here
PeterSurda commented 2022-03-09 09:39:23 +01:00
Review
Copy Link

I think you're right.

I think you're right.
print(
"Usage: python3 multibuild.py <jobname> <repository> <branch> <buildbotUrl> <os_codename>"
"Usage: python3 multibuild.py <buildbotUrl> <repository> <branch> "
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-09 07:43:38 +01:00
Outdated
Review
Copy Link

remove ".buildbot"

remove `".buildbot"`
👍 1
PeterSurda commented 2022-03-09 12:48:19 +01:00
Outdated
Review
Copy Link

This help message needs to be updated to reflect the arguments correctly.

This help message needs to be updated to reflect the arguments correctly.
👍 1
)
PeterSurda marked this conversation as resolved Outdated
PeterSurda commented 2022-03-08 07:56:35 +01:00
Outdated
Review
Copy Link

again remove jobname, is_build_script_available, is_test_script_available

again remove `jobname`, `is_build_script_available`, `is_test_script_available`
👍 1