Singularity Mesos Auth

[ghost]

Is there any possible way to authenticate to the mesos api? I made it, because I dont want that everyone can read or write the API from Mesos. Unfortunately Singularity cant authenticate too and I found no way to it. I tried it with credentialPrincipal and credentialSecret without success.

Also Iam searching for a way to do this with the Singularity API. Is there any way to protect the API with simple credentials?

[ssalinas]

I will have to investigate further on the credentialPrincipal/credentialSecret. There is a chance something went funky with the move to the http api there.

As for auth, we don't have a 'Basic Auth' solution built into singularity just yet (#19), but we do have a webhook auth similar to how kubernetes webhook token auth is set up. You can see docs on that here: http://getsingularity.com/Docs/features/auth.html

[ghost]

Alright, that sounds good. I also found out that if mesos runs on https the api calls from singularity go to http. Is there a way to declare it as https calls?
Specifically I mean the master port. In the API I saw that I can provide a httpsslave port.

[ghost]

Update: If I try to provide the credentials in the mesos section I get an nullpointer exception and if I provide it in the server section I get an "unknown response from master 401" so he can authenticate. I searched a bit more and found

I0321 07:32:47.414669 15743 zookeeper.cpp:262] A new leading master (UPID=master@xxx.xxx.xxx.xxx:5050) is detected
I0321 07:32:47.414726 15739 sched.cpp:336] New master detected at master@xxx.xxx.xxx.xxx:5050
I0321 07:32:47.414897 15739 sched.cpp:351] No credentials provided. Attempting to register without authentication
I0321 07:32:47.417392 15741 sched.cpp:751] Framework registered with Singularity

("The xxx in the ips are naturally from me")

So I dont know where and how I should write the credentials...

With https mesos and singularity I still get errors. He just make requests to the http api I think and not to the https. I would be great if I can configure it to https.

Update: I think the credentials are for the framework authentication and not for the mesos api Iam right or? Anyone can help?

This came when I write it to no namespace in the root from the second node. The first gets the error from top

I0321 07:42:52.017290 26301 authenticatee.cpp:97] Initializing client SASL
I0321 07:42:52.018304 26301 authenticatee.cpp:121] Creating new client SASL connection
I0321 07:42:52.019793 26305 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0321 07:42:52.019819 26305 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0321 07:42:52.020398 26302 authenticatee.cpp:259] Received SASL authentication step
I0321 07:42:52.020884 26300 authenticatee.cpp:299] Authentication success
I0321 07:42:52.025300 26300 sched.cpp:502] Successfully authenticated with master master@xxx.xxx.xxx.xxx:5050
I0321 07:42:52.026069 26305 sched.cpp:751] Framework registered with Singularity
ERROR [2018-03-21 06:42:52,130] com.hubspot.singularity.mesos.SingularityMesosSchedulerDelegator: Scheduler threw an uncaught exception - exiting
! com.hubspot.mesos.client.MesosClient$MesosClientException: Invalid response code from http://xxx.xxx.xxx.xxx:5050/master/state : 401
! at com.hubspot.mesos.client.SingularityMesosClient.getFromMesos(SingularityMesosClient.java:68)
! at com.hubspot.mesos.client.SingularityMesosClient.getFromMesos(SingularityMesosClient.java:75)
! at com.hubspot.mesos.client.SingularityMesosClient.getMasterState(SingularityMesosClient.java:86)
! at com.hubspot.singularity.mesos.SingularityStartup.startup(SingularityStartup.java:86)
! at com.hubspot.singularity.mesos.SingularityMesosSchedulerDelegator.startup(SingularityMesosSchedulerDelegator.java:112)

[ghost]

Still dont work.

If I try in Singularity 19.2 to provide the mesosUsername and mesosPassword I think following happens.
Singularity sends a request to http://username:password@http://xxx.xxx.xxx.xxx:5050/redirect and fails with name resolution. When I try it with a REST-Programm I fail too. If I change the url to http://username:password@xxx.xxx.xxx.xxx:5050/redirect it is successfull in the REST-Programm.
Down below he writes again

The logs:

INFO  [2018-04-03 06:03:15,608] org.apache.zookeeper.ClientCnxn: Socket connection established to xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx:2181, initiating session
INFO  [2018-04-03 06:03:15,614] org.apache.zookeeper.ClientCnxn: Session establishment complete on server xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx:2181, sessionid = 0x2624db268530029, negotiated timeout = 40000
INFO  [2018-04-03 06:03:15,617] org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
WARN  [2018-04-03 06:03:16,637] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Not running SingularityMailRecordCleaner due to delay value of 0
INFO  [2018-04-03 06:03:16,742] com.mesosphere.mesos.rx.java.MesosClient: Probing Mesos server at http://mesosUsername:mesosPassword@http://xxx.xxx.xxx.xxx:5050/redirect
ERROR [2018-04-03 06:03:17,159] com.hubspot.singularity.mesos.SingularityMesosSchedulerClient: Could not connect:
! java.net.UnknownHostException: http: Temporary failure in name resolution

INFO  [2018-04-03 06:03:17,166] com.mesosphere.mesos.rx.java.MesosClient: Probing Mesos server at http://mesosUsername:mesosPassword@xxx.xxx.xxx.xxx:5050/redirect
INFO  [2018-04-03 06:03:17,354] com.mesosphere.mesos.rx.java.MesosClient: Using new Mesos URI: http://xxx.xxx.xxx.xxx:5050/api/v1/scheduler
ERROR [2018-04-03 06:03:17,874] com.hubspot.singularity.mesos.SingularityMesosScheduler: Scheduler threw an uncaught exception - exiting
! com.hubspot.mesos.client.MesosClient$MesosClientException: Invalid response code from http://xxx.xxx.xxx.xxx:5050/master/state : 401

My Mesos Configuration Part:

mesos:
  master: http://xxx.xxx.xxx.xxx:5050,xxx.xxx.xxx.xxx:5050
  defaultCpus: 1
  defaultMemory: 128
  frameworkName: Singularity
  frameworkId: Singularity
  frameworkFailoverTimeout: 1000000
  mesosUsername: mesosUsername
  mesosPassword: mesosPassword

Still an issue or or did I make a mistake?

[ssalinas]

So, for the first error, one of your master urls has http on the front and one does not. I can fix the code to better check for that. As for the second error, I may need to file an issue with mesos-rxjava. Seems it does not carry over the credentials after being redirected to the active master.

[ghost]

Yes I think there is something strange. I tried it again without http and It still dont work. I dont know if I make some mistakes.

EDIT: I tried it again.. and now it works correctly. Dont know why dont know what I changed...
But then no issue need to be open at mesos-rxjava.

Would it be possible that providing the http before the master adresses is possible? So its easier to configure mesos https or is there another way?

[ssalinas]

Can you paste the format version that works correctly? One of our team will make a PR so our parsing handles more cases

[ghost]

mesos:
  master: xxx.xxx.xxx.xxx:5050,xxx.xxx.xxx.xxx:5050
  defaultCpus: 1
  defaultMemory: 128
  defaultDisk: 1024
  frameworkName: Singularity
  frameworkId: Singularity
  frameworkFailoverTimeout: 1000000
  mesosUsername: username
  mesosPassword: password

This is the mesos part in my configuration.

So you say the https part will come? Simple were it with http:ip:port/https:ip:port. Simple to configure and I hope simple to parse.

EDIT: I saw that it works only sometimes, so I searched for logs and found this:

INFO  [2018-04-10 05:29:23,882] com.mesosphere.mesos.rx.java.MesosClient: Probing Mesos server at http://username:password@xxx.xxx.xxx.xxx:5050/redirect
INFO  [2018-04-10 05:29:24,399] com.mesosphere.mesos.rx.java.MesosClient: Using new Mesos URI: http://xxx.xxx.xxx.xxx:5050/api/v1/scheduler

AND

WARN  [2018-04-10 05:32:11,175] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Not running SingularityMailRecordCleaner due to delay value of 0
INFO  [2018-04-10 05:32:11,293] com.mesosphere.mesos.rx.java.MesosClient: Probing Mesos server at http://username:password@xxx.xxx.xxx.xxx:5050/redirect
INFO  [2018-04-10 05:32:11,803] com.mesosphere.mesos.rx.java.MesosClient: Using new Mesos URI: http://xxx.xxx.xxx.xxx:5050/api/v1/scheduler
ERROR [2018-04-10 05:32:12,355] com.hubspot.singularity.mesos.SingularityMesosScheduler: Scheduler threw an uncaught exception - exiting
! com.hubspot.mesos.client.MesosClient$MesosClientException: Invalid response code from http://xxx.xxx.xxx.xxx:5050/master/state : 401
! at com.hubspot.mesos.client.SingularityMesosClient.getFromMesos(SingularityMesosClient.java:70)

When the authentication and the redirect is on the same server it works. But when it is on different it fail with an 401 unauthenticated. He authenticate only on one server and not at the other where the next request goes and fails.

[ssalinas]

Opened a PR with mesos-rxjava at d2iq-archive/mesos-rxjava#84
PR for simpler parsing here #1783

[ssalinas]

Fixed in #1867