Jul 27 20:01:59.745308 ip-10-0-138-123 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 20:01:59.745321 ip-10-0-138-123 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 20:01:59.745330 ip-10-0-138-123 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 20:01:59.745736 ip-10-0-138-123 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 20:02:09.780323 ip-10-0-138-123 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 20:02:09.780337 ip-10-0-138-123 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9bae2f3e4ff24b11b53a905ab81e7f66 --
Jul 27 20:03:53.803466 ip-10-0-138-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:03:54.149248 ip-10-0-138-123 kubenswrapper[2531]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:03:54.149248 ip-10-0-138-123 kubenswrapper[2531]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:03:54.149248 ip-10-0-138-123 kubenswrapper[2531]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:03:54.149248 ip-10-0-138-123 kubenswrapper[2531]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:03:54.149248 ip-10-0-138-123 kubenswrapper[2531]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:03:54.151048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.150930    2531 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:03:54.155264 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155248    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.155264 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155263    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155267    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155270    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155273    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155276    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155279    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155282    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155284    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155287    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155290    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155293    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155297    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155300    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155303    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155305    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155308    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155313    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155317    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155320    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155323    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.155329 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155326    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155330    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155333    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155337    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155340    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155343    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155346    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155348    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155351    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155354    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155356    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155359    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155362    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155364    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155367    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155370    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155373    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155375    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155378    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.155806 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155380    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155383    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155386    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155388    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155391    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155393    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155396    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155398    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155401    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155403    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155406    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155409    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155411    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155414    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155417    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155420    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155423    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155425    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155428    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.156292 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155430    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155435    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155439    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155442    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155445    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155447    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155450    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155453    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155455    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155458    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155461    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155463    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155466    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155468    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155471    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155474    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155476    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155479    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155482    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155484    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.156738 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155487    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155490    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155492    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155497    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155501    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155504    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155507    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155510    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155514    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155517    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155520    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155523    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155526    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155529    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155531    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155534    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155537    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155539    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155542    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.155545    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.157284 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156280    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156287    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156289    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156292    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156295    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156298    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156300    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156303    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156306    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156310    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156313    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156316    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156319    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156322    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156325    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156327    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156330    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156333    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156336    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156339    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.157760 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156342    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156345    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156347    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156350    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156353    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156355    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156358    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156361    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156363    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156366    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156368    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156371    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156374    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156377    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156379    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156382    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156385    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156387    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156389    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156392    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.158269 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156395    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156399    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156402    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156406    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156408    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156411    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156414    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156417    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156419    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156422    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156425    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156428    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156431    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156433    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156435    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156439    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156442    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156446    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156449    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.158763 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156452    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156455    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156458    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156460    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156463    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156466    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156469    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156471    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156474    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156478    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156480    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156483    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156486    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156488    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156491    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156494    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156497    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156499    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156501    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.159259 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156504    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156507    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156509    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156511    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156514    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156517    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156519    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156522    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156524    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156527    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156529    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156531    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156534    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156536    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156538    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156541    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156543    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156546    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156548    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156551    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.159711 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156553    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156922    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156927    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156930    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156932    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156935    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156938    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156941    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156943    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156946    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156948    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156951    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156954    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156957    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156959    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156962    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156966    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156969    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156972    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.160223 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156975    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156978    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156980    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156983    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156985    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156988    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156990    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156994    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.156998    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157001    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157005    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157008    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157010    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157016    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157019    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157022    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157025    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157028    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157030    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.160683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157034    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157036    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157039    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157042    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157044    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157047    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157050    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157052    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157055    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157057    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157060    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157062    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157065    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157069    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157072    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157074    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157077    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157080    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157082    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157085    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.161145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157088    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157091    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157093    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157096    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157098    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157101    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157105    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157108    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157110    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157113    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157115    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157119    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157123    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157126    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157128    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157131    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157134    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157137    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157139    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157142    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.161626 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157145    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157147    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157150    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157154    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157156    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157161    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157163    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157166    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157169    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157184    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157187    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157189    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157192    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157195    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157198    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157200    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157203    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157205    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157209    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157212    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.162159 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157215    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157217    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157289    2531 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157296    2531 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157302    2531 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157306    2531 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157310    2531 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157313    2531 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157317    2531 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157322    2531 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157325    2531 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157328    2531 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157331    2531 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157335    2531 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157338    2531 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157341    2531 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157344    2531 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157348    2531 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157352    2531 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157355    2531 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157358    2531 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157361    2531 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157364    2531 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157367    2531 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:03:54.162667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157371    2531 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157375    2531 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157378    2531 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157381    2531 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157384    2531 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157387    2531 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157390    2531 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157393    2531 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157396    2531 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157399    2531 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157404    2531 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157407    2531 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157410    2531 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157413    2531 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157416    2531 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157419    2531 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157423    2531 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157426    2531 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157429    2531 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157432    2531 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157435    2531 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157439    2531 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157441    2531 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157444    2531 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157448    2531 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:03:54.163286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157452    2531 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157455    2531 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157458    2531 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157462    2531 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157465    2531 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157468    2531 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157470    2531 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157474    2531 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157477    2531 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157480    2531 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157483    2531 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157485    2531 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157488    2531 flags.go:64] FLAG: --help="false"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157492    2531 flags.go:64] FLAG: --hostname-override="ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157495    2531 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157498    2531 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157501    2531 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157504    2531 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157508    2531 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157510    2531 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157513    2531 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157516    2531 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157519    2531 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:03:54.163878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157522    2531 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157526    2531 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157528    2531 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157531    2531 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157534    2531 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157537    2531 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157539    2531 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157542    2531 flags.go:64] FLAG: --lock-file=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157545    2531 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157549    2531 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157552    2531 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157557    2531 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157561    2531 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157563    2531 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157566    2531 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157569    2531 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157572    2531 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157575    2531 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157578    2531 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157582    2531 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157585    2531 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157589    2531 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157592    2531 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157595    2531 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157598    2531 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:03:54.164822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157601    2531 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157604    2531 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157608    2531 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157611    2531 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157618    2531 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157621    2531 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157624    2531 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157628    2531 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157630    2531 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157636    2531 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157639    2531 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157642    2531 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157645    2531 flags.go:64] FLAG: --port="10250"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157648    2531 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157651    2531 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d5443da549176151"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157655    2531 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157659    2531 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157662    2531 flags.go:64] FLAG: --register-node="true"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157665    2531 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157668    2531 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157672    2531 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157675    2531 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157677    2531 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157681    2531 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157684    2531 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:03:54.165848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157687    2531 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157690    2531 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157693    2531 flags.go:64] FLAG: --runonce="false"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157695    2531 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157698    2531 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157701    2531 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157704    2531 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157707    2531 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157710    2531 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157713    2531 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157716    2531 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157719    2531 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157722    2531 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157725    2531 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157727    2531 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157730    2531 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157733    2531 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157736    2531 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157741    2531 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157744    2531 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157747    2531 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157750    2531 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157753    2531 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157757    2531 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157760    2531 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:03:54.166961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157763    2531 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157766    2531 flags.go:64] FLAG: --v="2"
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157770    2531 flags.go:64] FLAG: --version="false"
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157774    2531 flags.go:64] FLAG: --vmodule=""
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157778    2531 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.157781    2531 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157895    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157899    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157902    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157905    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157908    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157911    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157915    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157918    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157921    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157923    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157926    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157930    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157934    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157936    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157939    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.167751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157942    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157945    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157949    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157952    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157955    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157958    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157961    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157964    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157966    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157970    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157973    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157975    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157978    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157981    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157984    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157986    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157989    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157992    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157994    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.157997    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.168294 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158000    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158002    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158006    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158011    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158014    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158016    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158019    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158021    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158024    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158027    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158029    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158032    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158034    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158037    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158040    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158042    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158045    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158047    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158050    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.168801 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158052    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158055    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158059    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158061    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158064    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158067    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158069    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158072    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158074    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158077    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158080    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158082    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158085    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158087    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158090    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158092    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158096    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158098    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158101    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158104    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.169282 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158106    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158109    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158111    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158114    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158117    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158119    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158122    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158125    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158128    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158130    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158133    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158136    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158138    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158141    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158145    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158147    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158150    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158152    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158155    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158157    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.169753 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158160    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158163    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158166    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158168    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.158183    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.158191    2531 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.167642    2531 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.167658    2531 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.168333    2531 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.168341    2531 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168402    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168407    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168410    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168413    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168416    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.170283 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168419    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168422    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168425    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168428    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168431    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168434    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168436    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168439    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168442    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168446    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168450    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168453    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168463    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168466    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168469    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168472    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168475    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168478    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168481    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.170677 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168483    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168486    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168489    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168492    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168494    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168498    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168501    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168505    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168509    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168512    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168515    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168518    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168520    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168523    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168525    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168528    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168531    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168533    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168536    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168538    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.171167 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168541    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168544    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168547    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168550    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168553    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168555    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168558    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168560    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168563    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168565    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168568    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168571    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168573    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168577    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168580    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168583    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168586    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168590    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168594    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168597    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.171653 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168599    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168602    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168605    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168607    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168610    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168612    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168615    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168618    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168620    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168623    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168626    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168629    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168631    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168634    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168636    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168640    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168642    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168645    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168648    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.172115 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168650    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168653    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168656    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168659    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168661    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168664    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168666    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168669    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168672    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168674    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168678    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168680    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168683    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168685    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168688    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.172768 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168691    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.168696    2531 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168830    2531 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168836    2531 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168839    2531 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168842    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168845    2531 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168847    2531 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168850    2531 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168853    2531 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168855    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168858    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168862    2531 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168866    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168869    2531 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:03:54.173120 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168872    2531 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168875    2531 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168878    2531 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168880    2531 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168883    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168886    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168889    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168892    2531 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168900    2531 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168909    2531 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168912    2531 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168914    2531 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168917    2531 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168920    2531 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168923    2531 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168925    2531 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168928    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168931    2531 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168934    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168936    2531 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:03:54.173512 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168939    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168942    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168944    2531 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168947    2531 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168949    2531 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168952    2531 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168954    2531 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168957    2531 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168960    2531 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168962    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168965    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168968    2531 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168971    2531 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168974    2531 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168976    2531 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168979    2531 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168982    2531 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168985    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168987    2531 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:03:54.174015 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168990    2531 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168992    2531 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168995    2531 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.168998    2531 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169000    2531 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169003    2531 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169006    2531 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169009    2531 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169011    2531 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169014    2531 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169017    2531 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169020    2531 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169022    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169025    2531 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169027    2531 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169030    2531 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169032    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169035    2531 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169037    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169040    2531 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:03:54.174491 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169043    2531 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169047    2531 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169050    2531 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169053    2531 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169056    2531 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169072    2531 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169076    2531 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169079    2531 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169082    2531 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169084    2531 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169087    2531 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169089    2531 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169092    2531 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169094    2531 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169097    2531 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169099    2531 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169102    2531 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169104    2531 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169107    2531 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:03:54.174990 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169109    2531 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169112    2531 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169114    2531 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169116    2531 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169119    2531 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169122    2531 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169124    2531 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:54.169126    2531 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.169131    2531 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.169258    2531 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.173448    2531 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.174864    2531 server.go:1019] "Starting client certificate rotation"
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.174979    2531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:03:54.175461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.175023    2531 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:03:54.177361 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.177347    2531 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:03:54.178544 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.178531    2531 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:03:54.191544 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.191521    2531 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:03:54.194304 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.194286    2531 log.go:25] "Validated CRI v1 image API"
Jul 27 20:03:54.195128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.195110    2531 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:03:54.197465 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.197441    2531 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e16bcc56-0041-4057-bba1-f21d0f6429c6:/dev/nvme0n1p3 fe78a7ea-e35f-4a02-93bf-f63dbabce519:/dev/nvme0n1p4]
Jul 27 20:03:54.197550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.197461    2531 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:03:54.202425 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.202305    2531 manager.go:216] Machine: {Timestamp:2026-07-27 20:03:54.20115564 +0000 UTC m=+0.295741094 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3141733 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20e6289c2432f5292578b52850f56a SystemUUID:ec20e628-9c24-32f5-2925-78b52850f56a BootID:9bae2f3e-4ff2-4b11-b53a-905ab81e7f66 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:ee:bc:89:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:ee:bc:89:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:1c:a1:99:34:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:03:54.202425 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.202413    2531 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:03:54.202550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.202491    2531 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:03:54.203628 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.203612    2531 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:03:54.205384 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.205360    2531 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:03:54.205528 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.205388    2531 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:03:54.206093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.206083    2531 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:03:54.206122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.206095    2531 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:03:54.206122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.206106    2531 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:03:54.206823 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.206813    2531 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:03:54.207711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.207700    2531 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:03:54.210391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210379    2531 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:03:54.210539 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210529    2531 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:03:54.210579 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210560    2531 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:03:54.210579 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210569    2531 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:03:54.210579 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210579    2531 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:03:54.210692 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210589    2531 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:03:54.210692 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.210598    2531 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:03:54.211592 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.211569    2531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:03:54.211682 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.211603    2531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:03:54.213947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.213930    2531 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:03:54.215974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.215961    2531 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:03:54.216031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.215990    2531 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:03:54.217617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217605    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217621    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217628    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217633    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217642    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217649    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217657    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:03:54.217664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217662    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:03:54.217856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217670    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:03:54.217856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217675    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:03:54.217856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217686    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:03:54.217856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.217692    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:03:54.219055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.219042    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:03:54.219055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.219051    2531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:03:54.222204 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.222189    2531 server.go:1268] "Started kubelet"
Jul 27 20:03:54.222286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.222255    2531 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:03:54.222443 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.222391    2531 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:03:54.222653 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.222636    2531 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:03:54.222838 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.222448    2531 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:03:54.222908 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.222548    2531 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:03:54.222908 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.222440    2531 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:03:54.223102 ip-10-0-138-123 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:03:54.224341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.224329    2531 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:03:54.226222 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.226206    2531 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:03:54.227379 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.227357    2531 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:03:54.232288 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.230703    2531 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-123.ec2.internal.18c63e25c401ccd8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-123.ec2.internal,UID:ip-10-0-138-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-123.ec2.internal,},FirstTimestamp:2026-07-27 20:03:54.222144728 +0000 UTC m=+0.316730184,LastTimestamp:2026-07-27 20:03:54.222144728 +0000 UTC m=+0.316730184,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-123.ec2.internal,}"
Jul 27 20:03:54.233203 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233151    2531 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:03:54.233329 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233155    2531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:03:54.233885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233810    2531 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:03:54.233885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233820    2531 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:03:54.233885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233823    2531 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:03:54.234030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233961    2531 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:03:54.234030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.233970    2531 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:03:54.234332 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.234311    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.234744 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.234724    2531 factory.go:55] Registering systemd factory
Jul 27 20:03:54.234811 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.234766    2531 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:03:54.235010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.234995    2531 factory.go:153] Registering CRI-O factory
Jul 27 20:03:54.235081 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.235013    2531 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:03:54.235081 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.235064    2531 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:03:54.235164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.235087    2531 factory.go:103] Registering Raw factory
Jul 27 20:03:54.235164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.235105    2531 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:03:54.235368 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.235349    2531 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 20:03:54.235684 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.235671    2531 manager.go:349] Starting recovery of all containers
Jul 27 20:03:54.235751 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.235663    2531 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 20:03:54.245203 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.245159    2531 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qnvk9"
Jul 27 20:03:54.246313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.246294    2531 manager.go:354] Recovery completed
Jul 27 20:03:54.250601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.250468    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.252452 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.252434    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.252534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.252466    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.252534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.252479    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.253102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253086    2531 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:03:54.253102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253101    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.253220 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253103    2531 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:03:54.253220 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253186    2531 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:03:54.253421 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253406    2531 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qnvk9"
Jul 27 20:03:54.253753 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253738    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.253806 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253763    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.253806 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.253775    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.255843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.255828    2531 policy_none.go:49] "None policy: Start"
Jul 27 20:03:54.255843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.255845    2531 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:03:54.255932 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.255856    2531 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:03:54.257818 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.257802    2531 policy_none.go:47] "Start"
Jul 27 20:03:54.287804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.287783    2531 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:03:54.287921 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.287825    2531 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:03:54.287921 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.287840    2531 server.go:87] "Starting device plugin registration server"
Jul 27 20:03:54.288097 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.288086    2531 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:03:54.288140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.288100    2531 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:03:54.288225 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.288207    2531 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:03:54.288296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.288286    2531 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:03:54.288296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.288296    2531 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:03:54.289458 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.289437    2531 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:03:54.289549 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.289492    2531 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.330673 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.330641    2531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:03:54.338296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.331911    2531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:03:54.338296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.331946    2531 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:03:54.338296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.331981    2531 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:03:54.338296 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.332014    2531 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:03:54.338296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.334381    2531 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:03:54.388493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.388458    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.389459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.389442    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.389567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.389475    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.389567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.389490    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.389567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.389518    2531 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.396215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.396198    2531 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.396297 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.396227    2531 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-123.ec2.internal\": node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.414634 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.414581    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.432946 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.432913    2531 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"]
Jul 27 20:03:54.433008 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.432990    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.434004 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.433989    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.434061 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.434018    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.434061 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.434033    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.435115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.435097    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.435162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.435124    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.436339 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.436325    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.436464 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.436449    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.436530 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.436479    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.437262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437247    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.437347 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437278    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.437347 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437288    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.437347 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437245    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.437347 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437340    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.437496 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.437357    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.439732 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.439715    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.439795 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.439750    2531 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:03:54.440928 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.440915    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:03:54.441017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.440943    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:03:54.441017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.440957    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:03:54.461849 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.461828    2531 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.466478 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.466462    2531 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.514960 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.514927    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.536264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.536238    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.536318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.536279    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.536318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.536299    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec91069f1c74047cfed254ac36b1a990-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"ec91069f1c74047cfed254ac36b1a990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.536382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.536329    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.536382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.536344    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f59d7fb7872ccb4c615dc2b4549c54-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"30f59d7fb7872ccb4c615dc2b4549c54\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.615751 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.615717    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.637146 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.637116    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec91069f1c74047cfed254ac36b1a990-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"ec91069f1c74047cfed254ac36b1a990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.637227 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.637163    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec91069f1c74047cfed254ac36b1a990-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"ec91069f1c74047cfed254ac36b1a990\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.716638 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.716564    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.766138 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.766110    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.769762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:54.769741    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:54.817474 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.817438    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:54.924129 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:54.924078    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.024674 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.024592    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.126548 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.126509    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.175671 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.175640    2531 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:03:55.176160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.175802    2531 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:03:55.215658 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.215632    2531 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:03:55.226641 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.226611    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.260055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.259996    2531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:58:55 +0000 UTC" deadline="2028-02-26 13:56:05.742949307 +0000 UTC"
Jul 27 20:03:55.260055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.260054    2531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13889h52m10.482899711s"
Jul 27 20:03:55.327407 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.327333    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.428064 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.428033    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.496683 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:55.496642    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f59d7fb7872ccb4c615dc2b4549c54.slice/crio-48d068171fa33480cdcc621bc0ff539776586fb17c7f5485d664b3e82e50c503 WatchSource:0}: Error finding container 48d068171fa33480cdcc621bc0ff539776586fb17c7f5485d664b3e82e50c503: Status 404 returned error can't find the container with id 48d068171fa33480cdcc621bc0ff539776586fb17c7f5485d664b3e82e50c503
Jul 27 20:03:55.508413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.508395    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:03:55.516430 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:55.516405    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec91069f1c74047cfed254ac36b1a990.slice/crio-4320942eb571951735af9362321f3d3482866573b2d540c8ec2ecd926b6767f2 WatchSource:0}: Error finding container 4320942eb571951735af9362321f3d3482866573b2d540c8ec2ecd926b6767f2: Status 404 returned error can't find the container with id 4320942eb571951735af9362321f3d3482866573b2d540c8ec2ecd926b6767f2
Jul 27 20:03:55.529036 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.529018    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.632860 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:55.632746    2531 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 27 20:03:55.672244 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.672218    2531 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:03:55.694185 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.694150    2531 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:03:55.734501 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.734462    2531 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:55.748532 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.748500    2531 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:03:55.749635 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.749620    2531 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 27 20:03:55.761701 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.761675    2531 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:03:55.981233 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:55.981152    2531 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:03:56.211869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.211837    2531 apiserver.go:52] "Watching apiserver"
Jul 27 20:03:56.217969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.217936    2531 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:03:56.218468 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.218423    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dflm4","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs","openshift-image-registry/node-ca-qd678","openshift-multus/multus-h698x","openshift-multus/network-metrics-daemon-c9krr","openshift-network-diagnostics/network-check-target-fgn6x","openshift-network-operator/iptables-alerter-dvkf6","kube-system/konnectivity-agent-rkvwd","openshift-cluster-node-tuning-operator/tuned-zbmqq","openshift-dns/node-resolver-f27hf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-4dkm5","openshift-ovn-kubernetes/ovnkube-node-6wqxm"]
Jul 27 20:03:56.221284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.221259    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:56.221400 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.221357    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:03:56.223717 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.223694    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.226145 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.226120    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.226920 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.226901    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bkhpj\""
Jul 27 20:03:56.227019 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.226918    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:03:56.227019 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.226954    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:03:56.227136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.227083    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.227136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.227087    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:03:56.227265 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.227233    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.228542 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.228518    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.228919 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.228898    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.229181 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.229161    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qnnjz\""
Jul 27 20:03:56.229272 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.229177    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:03:56.229422 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.229407    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.231125 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.231104    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.231722 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.231675    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.231722 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.231674    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:03:56.231868 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.231755    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4m4g4\""
Jul 27 20:03:56.231923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.231911    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.233266 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.233234    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mz4gq\""
Jul 27 20:03:56.233350 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.233303    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:03:56.233750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.233732    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.233818 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.233800    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:03:56.234904 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.234884    2531 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:03:56.236669 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.236592    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.238842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.238821    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:03:56.239048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.239030    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:03:56.243275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.240323    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ffgf6\""
Jul 27 20:03:56.243275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.241492    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.243275 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.241575    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:03:56.244850 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.244824    2531 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:03:56.245303 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245169    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.245675 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245646    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-device-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.245764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245694    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.245764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245720    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.245872 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245805    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-multus\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.245872 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245843    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxhx\" (UniqueName: \"kubernetes.io/projected/6ab5f088-3f79-4187-9206-27a21a968b13-kube-api-access-gmxhx\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.245974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245870    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.245974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245897    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-os-release\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.245974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245924    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-bin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.245980    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.246128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246016    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.246128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246044    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.246128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246072    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtjd\" (UniqueName: \"kubernetes.io/projected/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-kube-api-access-nrtjd\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.246128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246100    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-socket-dir-parent\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246137    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-kubelet\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246205    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-multus-certs\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246265    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246298    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246321    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/343e6847-03ce-4a40-96ea-0e0524a2110e-host\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.246392 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246354    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47b5\" (UniqueName: \"kubernetes.io/projected/343e6847-03ce-4a40-96ea-0e0524a2110e-kube-api-access-t47b5\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246380    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdg9\" (UniqueName: \"kubernetes.io/projected/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-kube-api-access-tcdg9\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246470    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-system-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246499    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-cni-binary-copy\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246522    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-etc-kubernetes\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246548    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad281922-6789-47b3-9d99-8920f6816fef-agent-certs\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246575    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246605    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246633    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-system-cni-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246658    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cnibin\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.246694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246688    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246711    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-conf-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246748    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-multus-daemon-config\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246792    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad281922-6789-47b3-9d99-8920f6816fef-konnectivity-ca\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246822    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246847    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/343e6847-03ce-4a40-96ea-0e0524a2110e-serviceca\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246881    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-netns\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246936    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxsd\" (UniqueName: \"kubernetes.io/projected/56d5077a-f097-4203-8c38-51963fabd5f3-kube-api-access-8nxsd\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.246994    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-os-release\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.247026    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-cnibin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.247051    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-k8s-cni-cncf-io\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247171 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.247118    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-hostroot\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.247758 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.247349    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:03:56.247758 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.247688    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.248712 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.248689    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:03:56.248849 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.248753    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8fvm6\""
Jul 27 20:03:56.248966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.248947    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.249029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.248991    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:03:56.249165 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.249149    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:03:56.249273 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.249239    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.249806 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.249785    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.249892 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.249855    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9lpf6\""
Jul 27 20:03:56.250100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.250076    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.251540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.251517    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.254200 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.254182    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.254854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.254578    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.254854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.254602    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mxzhx\""
Jul 27 20:03:56.254854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.254830    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.257021 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.256997    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:03:56.257283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.257264    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-89djn\""
Jul 27 20:03:56.257370 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.257309    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:03:56.257437 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.257310    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:03:56.276931 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.276433    2531 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-snpn4"
Jul 27 20:03:56.286448 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.286421    2531 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-snpn4"
Jul 27 20:03:56.335833 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.335797    2531 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:03:56.341714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.341651    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"ec91069f1c74047cfed254ac36b1a990","Type":"ContainerStarted","Data":"4320942eb571951735af9362321f3d3482866573b2d540c8ec2ecd926b6767f2"}
Jul 27 20:03:56.343447 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.343418    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"30f59d7fb7872ccb4c615dc2b4549c54","Type":"ContainerStarted","Data":"48d068171fa33480cdcc621bc0ff539776586fb17c7f5485d664b3e82e50c503"}
Jul 27 20:03:56.347364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347298    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-kubelet\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.347364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347343    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-system-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347371    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad281922-6789-47b3-9d99-8920f6816fef-agent-certs\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347397    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347424    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-ovn\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347448    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-host\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347473    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347497    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-tmp-dir\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347523    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.347563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347547    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-env-overrides\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347573    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-script-lib\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347597    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-modprobe-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347625    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxsd\" (UniqueName: \"kubernetes.io/projected/56d5077a-f097-4203-8c38-51963fabd5f3-kube-api-access-8nxsd\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347649    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-etc-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347676    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347703    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-427x6\" (UniqueName: \"kubernetes.io/projected/d881ed48-6c18-454c-83e4-6dda56328c2f-kube-api-access-427x6\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347747    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-k8s-cni-cncf-io\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347776    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-device-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347802    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347844    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347873    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347907    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-bin\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347933    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft6w\" (UniqueName: \"kubernetes.io/projected/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-kube-api-access-rft6w\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.347962    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxhx\" (UniqueName: \"kubernetes.io/projected/6ab5f088-3f79-4187-9206-27a21a968b13-kube-api-access-gmxhx\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.348009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348009    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-var-lib-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348035    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-log-socket\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348060    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-config\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348088    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-bin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348114    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348146    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-node-log\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348172    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysconfig\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348195    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-tmp\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348197    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-system-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348223    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-multus-certs\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348270    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348298    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47b5\" (UniqueName: \"kubernetes.io/projected/343e6847-03ce-4a40-96ea-0e0524a2110e-kube-api-access-t47b5\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348331    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-var-lib-kubelet\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348358    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdg9\" (UniqueName: \"kubernetes.io/projected/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-kube-api-access-tcdg9\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348373    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-sys-fs\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348383    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-dbus\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348421    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-lib-modules\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.348724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348457    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/343e6847-03ce-4a40-96ea-0e0524a2110e-serviceca\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348623    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348630    2531 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348836    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-bin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348869    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-k8s-cni-cncf-io\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348907    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-hosts-file\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348937    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-cni-binary-copy\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348983    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-etc-kubernetes\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349015    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349044    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-system-cni-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349048    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-multus-certs\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349079    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cnibin\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349109    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-systemd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349153    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-conf-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349181    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-multus-daemon-config\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349209    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad281922-6789-47b3-9d99-8920f6816fef-konnectivity-ca\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349234    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2732a934-0b45-4204-b0f4-677985392b1a-host-slash\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349354    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-slash\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.349427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349382    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovn-node-metrics-cert\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349412    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-kubernetes\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349440    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-netns\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349468    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-os-release\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349493    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-netns\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349520    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-conf\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349547    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-cnibin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349573    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-hostroot\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349596    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-cni-binary-copy\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349596    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349625    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4gt\" (UniqueName: \"kubernetes.io/projected/2732a934-0b45-4204-b0f4-677985392b1a-kube-api-access-8p4gt\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349657    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-systemd-units\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349682    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349696    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-registration-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349723    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-device-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.348983    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-socket-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349794    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-system-cni-dir\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349832    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-hostroot\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349840    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-os-release\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349873    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-run\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349899    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-sys\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349929    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-multus\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349958    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349987    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-kubelet-config\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.349995    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-cnibin\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350015    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgpj\" (UniqueName: \"kubernetes.io/projected/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-kube-api-access-7cgpj\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350054    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-run-netns\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350087    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-cni-multus\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350086    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-systemd\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350162    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-os-release\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350215    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350221    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ab5f088-3f79-4187-9206-27a21a968b13-multus-daemon-config\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350248    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350281    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtjd\" (UniqueName: \"kubernetes.io/projected/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-kube-api-access-nrtjd\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350319    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/343e6847-03ce-4a40-96ea-0e0524a2110e-serviceca\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.350971 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.350364    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350366    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-cni-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350414    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cnibin\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350418    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-conf-dir\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350461    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-etc-kubernetes\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350475    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-os-release\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350475    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad281922-6789-47b3-9d99-8920f6816fef-konnectivity-ca\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350501    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350528    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2732a934-0b45-4204-b0f4-677985392b1a-iptables-alerter-script\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350573    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56d5077a-f097-4203-8c38-51963fabd5f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.350579    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:03:56.85053069 +0000 UTC m=+2.945116160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350704    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-netd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350755    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-socket-dir-parent\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350835    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-multus-socket-dir-parent\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350879    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-kubelet\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350922    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ab5f088-3f79-4187-9206-27a21a968b13-host-var-lib-kubelet\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.351938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.350974    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351022    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351040    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/343e6847-03ce-4a40-96ea-0e0524a2110e-host\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351159    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/343e6847-03ce-4a40-96ea-0e0524a2110e-host\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351153    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351225    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-tuned\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351265    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.352721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.351873    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.354149 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.354124    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad281922-6789-47b3-9d99-8920f6816fef-agent-certs\") pod \"konnectivity-agent-rkvwd\" (UID: \"ad281922-6789-47b3-9d99-8920f6816fef\") " pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.357852 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.357771    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxhx\" (UniqueName: \"kubernetes.io/projected/6ab5f088-3f79-4187-9206-27a21a968b13-kube-api-access-gmxhx\") pod \"multus-h698x\" (UID: \"6ab5f088-3f79-4187-9206-27a21a968b13\") " pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.358771 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.358743    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxsd\" (UniqueName: \"kubernetes.io/projected/56d5077a-f097-4203-8c38-51963fabd5f3-kube-api-access-8nxsd\") pod \"aws-ebs-csi-driver-node-s2ggs\" (UID: \"56d5077a-f097-4203-8c38-51963fabd5f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.361713 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.361686    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtjd\" (UniqueName: \"kubernetes.io/projected/1b228cf7-a5ab-410f-b58b-1bcc0008a19f-kube-api-access-nrtjd\") pod \"multus-additional-cni-plugins-4dkm5\" (UID: \"1b228cf7-a5ab-410f-b58b-1bcc0008a19f\") " pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.361830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.361686    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47b5\" (UniqueName: \"kubernetes.io/projected/343e6847-03ce-4a40-96ea-0e0524a2110e-kube-api-access-t47b5\") pod \"node-ca-qd678\" (UID: \"343e6847-03ce-4a40-96ea-0e0524a2110e\") " pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.362085 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.362053    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:03:56.362159 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.362089    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:03:56.362159 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.362101    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:56.362335 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.362168    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:03:56.862152456 +0000 UTC m=+2.956737912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:56.363534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.363510    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdg9\" (UniqueName: \"kubernetes.io/projected/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-kube-api-access-tcdg9\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.452117 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452074    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2732a934-0b45-4204-b0f4-677985392b1a-host-slash\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.452117 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452122    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-slash\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452146    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovn-node-metrics-cert\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452169    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-kubernetes\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452192    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-netns\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452217    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2732a934-0b45-4204-b0f4-677985392b1a-host-slash\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452230    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-conf\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452237    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-slash\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452257    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4gt\" (UniqueName: \"kubernetes.io/projected/2732a934-0b45-4204-b0f4-677985392b1a-kube-api-access-8p4gt\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452279    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-systemd-units\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452300    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452320    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-kubernetes\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452442    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-conf\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452495    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-systemd-units\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452538    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-run\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452563    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-sys\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452606    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-kubelet-config\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452632    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgpj\" (UniqueName: \"kubernetes.io/projected/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-kube-api-access-7cgpj\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452655    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-systemd\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452683    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2732a934-0b45-4204-b0f4-677985392b1a-iptables-alerter-script\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452700    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-sys\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452710    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-netd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452445    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysctl-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452747    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-netns\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452784    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452796    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-kubelet-config\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452832    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-systemd\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452839    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-run\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452749    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-run-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.452994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452876    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-tuned\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452882    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-netd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452901    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-kubelet\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452940    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-ovn\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452934    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-kubelet\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452968    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-host\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.452999    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-tmp-dir\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453015    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-host\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453027    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453052    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-env-overrides\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453079    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-script-lib\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453104    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-modprobe-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453131    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-etc-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453157    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453186    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-427x6\" (UniqueName: \"kubernetes.io/projected/d881ed48-6c18-454c-83e4-6dda56328c2f-kube-api-access-427x6\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453220    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453246    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-bin\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.453890 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453286    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft6w\" (UniqueName: \"kubernetes.io/projected/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-kube-api-access-rft6w\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453314    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-var-lib-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453344    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-log-socket\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453369    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-config\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453395    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-node-log\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453418    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysconfig\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453441    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-tmp\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453468    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-var-lib-kubelet\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453492    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-dbus\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453515    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-lib-modules\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453533    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453589    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-hosts-file\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453590    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-tmp-dir\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453619    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-modprobe-d\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453623    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2732a934-0b45-4204-b0f4-677985392b1a-iptables-alerter-script\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453540    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-hosts-file\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453668    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-node-log\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453676    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-var-lib-kubelet\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.454820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453684    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-systemd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453706    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-etc-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453747    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fae3a580-5781-48b4-a8bb-8880bdcf81da-dbus\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453758    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-sysconfig\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453797    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-systemd\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453843    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-cni-bin\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453850    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d881ed48-6c18-454c-83e4-6dda56328c2f-lib-modules\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453844    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-script-lib\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453886    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-var-lib-openvswitch\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.453902    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.454179    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.454244    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:03:56.954225259 +0000 UTC m=+3.048810704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.454461    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-run-ovn\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.454512    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-log-socket\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.454831    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-env-overrides\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.455062    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovnkube-config\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.455827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.455722    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-ovn-node-metrics-cert\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.456702 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.456428    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-tmp\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.456991 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.456967    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d881ed48-6c18-454c-83e4-6dda56328c2f-etc-tuned\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.462523 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.462459    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgpj\" (UniqueName: \"kubernetes.io/projected/3dd1f022-a6a1-48fa-acaf-d545c56e8ad6-kube-api-access-7cgpj\") pod \"node-resolver-f27hf\" (UID: \"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6\") " pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.462949 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.462921    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4gt\" (UniqueName: \"kubernetes.io/projected/2732a934-0b45-4204-b0f4-677985392b1a-kube-api-access-8p4gt\") pod \"iptables-alerter-dvkf6\" (UID: \"2732a934-0b45-4204-b0f4-677985392b1a\") " pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.464112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.464081    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft6w\" (UniqueName: \"kubernetes.io/projected/3bc8a0d9-73c6-418b-9e23-30a52d74e92f-kube-api-access-rft6w\") pod \"ovnkube-node-6wqxm\" (UID: \"3bc8a0d9-73c6-418b-9e23-30a52d74e92f\") " pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.464474 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.464447    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-427x6\" (UniqueName: \"kubernetes.io/projected/d881ed48-6c18-454c-83e4-6dda56328c2f-kube-api-access-427x6\") pod \"tuned-zbmqq\" (UID: \"d881ed48-6c18-454c-83e4-6dda56328c2f\") " pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.538529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.538437    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4dkm5"
Jul 27 20:03:56.550115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.550080    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs"
Jul 27 20:03:56.560404 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.560374    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qd678"
Jul 27 20:03:56.566542 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.566517    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h698x"
Jul 27 20:03:56.575505 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.575480    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:03:56.584560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.584531    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:03:56.593710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.593686    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zbmqq"
Jul 27 20:03:56.603714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.603689    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f27hf"
Jul 27 20:03:56.613675 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.613648    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvkf6"
Jul 27 20:03:56.857242 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.857143    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:56.857420 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.857310    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:56.857420 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.857373    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:03:57.857359147 +0000 UTC m=+3.951944595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:56.958413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.958368    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:56.958413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:56.958425    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:56.958673 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958545    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:56.958673 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958591    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:03:56.958673 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958618    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:03:56.958673 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958627    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:03:57.958608733 +0000 UTC m=+4.053194179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:56.958673 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958631    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:56.958876 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:56.958690    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:03:57.958672792 +0000 UTC m=+4.053258258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:57.290865 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.290773    2531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:58:56 +0000 UTC" deadline="2027-12-28 21:00:32.811042179 +0000 UTC"
Jul 27 20:03:57.290865 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.290820    2531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12456h56m35.520225734s"
Jul 27 20:03:57.333190 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.333143    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:57.333367 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.333293    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:03:57.639103 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.639069    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b228cf7_a5ab_410f_b58b_1bcc0008a19f.slice/crio-f7ea30f79f379b68b4cfda1dbc82e37a8af3fd0df1065860902fcc0d6369a676 WatchSource:0}: Error finding container f7ea30f79f379b68b4cfda1dbc82e37a8af3fd0df1065860902fcc0d6369a676: Status 404 returned error can't find the container with id f7ea30f79f379b68b4cfda1dbc82e37a8af3fd0df1065860902fcc0d6369a676
Jul 27 20:03:57.641193 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.641128    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343e6847_03ce_4a40_96ea_0e0524a2110e.slice/crio-3200f17797ac846357e74427bfa2f8581e925a3cb452acd95211a387d028bcfd WatchSource:0}: Error finding container 3200f17797ac846357e74427bfa2f8581e925a3cb452acd95211a387d028bcfd: Status 404 returned error can't find the container with id 3200f17797ac846357e74427bfa2f8581e925a3cb452acd95211a387d028bcfd
Jul 27 20:03:57.642145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.642114    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd881ed48_6c18_454c_83e4_6dda56328c2f.slice/crio-ad499327fa85f802ceb3c34d353bdbbb75a4e435d7b9ca3c395b4a7b3e55c215 WatchSource:0}: Error finding container ad499327fa85f802ceb3c34d353bdbbb75a4e435d7b9ca3c395b4a7b3e55c215: Status 404 returned error can't find the container with id ad499327fa85f802ceb3c34d353bdbbb75a4e435d7b9ca3c395b4a7b3e55c215
Jul 27 20:03:57.645776 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.645746    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d5077a_f097_4203_8c38_51963fabd5f3.slice/crio-6cd172bcbc8d9deb7ab6d5e8efeee250c395dfff6358a14f9f397fdc1d074faf WatchSource:0}: Error finding container 6cd172bcbc8d9deb7ab6d5e8efeee250c395dfff6358a14f9f397fdc1d074faf: Status 404 returned error can't find the container with id 6cd172bcbc8d9deb7ab6d5e8efeee250c395dfff6358a14f9f397fdc1d074faf
Jul 27 20:03:57.646520 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.646439    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc8a0d9_73c6_418b_9e23_30a52d74e92f.slice/crio-db78e8b9613587787b5e114995f93da3f61e6056b159dbe31ff0095a29351a11 WatchSource:0}: Error finding container db78e8b9613587787b5e114995f93da3f61e6056b159dbe31ff0095a29351a11: Status 404 returned error can't find the container with id db78e8b9613587787b5e114995f93da3f61e6056b159dbe31ff0095a29351a11
Jul 27 20:03:57.648199 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.648172    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2732a934_0b45_4204_b0f4_677985392b1a.slice/crio-98f34e5fd7a14a6503887c867c9b13f6470d101ba816daca3dbc0d7b9fb5a424 WatchSource:0}: Error finding container 98f34e5fd7a14a6503887c867c9b13f6470d101ba816daca3dbc0d7b9fb5a424: Status 404 returned error can't find the container with id 98f34e5fd7a14a6503887c867c9b13f6470d101ba816daca3dbc0d7b9fb5a424
Jul 27 20:03:57.649129 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.648702    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab5f088_3f79_4187_9206_27a21a968b13.slice/crio-c839f84fe75237f83c02bafc983afac81542360cb1318313f9649568bace61e9 WatchSource:0}: Error finding container c839f84fe75237f83c02bafc983afac81542360cb1318313f9649568bace61e9: Status 404 returned error can't find the container with id c839f84fe75237f83c02bafc983afac81542360cb1318313f9649568bace61e9
Jul 27 20:03:57.650612 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.650585    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad281922_6789_47b3_9d99_8920f6816fef.slice/crio-3722aa2dcbe78182cef6e6dc80d91ce69db807f5b568661e63f1bcb68e0a2284 WatchSource:0}: Error finding container 3722aa2dcbe78182cef6e6dc80d91ce69db807f5b568661e63f1bcb68e0a2284: Status 404 returned error can't find the container with id 3722aa2dcbe78182cef6e6dc80d91ce69db807f5b568661e63f1bcb68e0a2284
Jul 27 20:03:57.651013 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:03:57.650989    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd1f022_a6a1_48fa_acaf_d545c56e8ad6.slice/crio-afed3099cc9eec2919a307f87ebe896b7a875f2dbc6520562a14ee271e4cd60b WatchSource:0}: Error finding container afed3099cc9eec2919a307f87ebe896b7a875f2dbc6520562a14ee271e4cd60b: Status 404 returned error can't find the container with id afed3099cc9eec2919a307f87ebe896b7a875f2dbc6520562a14ee271e4cd60b
Jul 27 20:03:57.866080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.865903    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:57.866080 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.866070    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:57.866284 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.866144    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:03:59.866125676 +0000 UTC m=+5.960711125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:57.967345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.967307    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:57.967358    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967460    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967482    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967503    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967513    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:57.967518 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967520    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:03:59.967502785 +0000 UTC m=+6.062088250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:57.967709 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:57.967556    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:03:59.96754351 +0000 UTC m=+6.062128971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:58.292529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.292318    2531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:58:56 +0000 UTC" deadline="2028-04-19 17:52:15.114704902 +0000 UTC"
Jul 27 20:03:58.292529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.292394    2531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15165h48m16.822315286s"
Jul 27 20:03:58.333926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.333891    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:58.334141 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:58.334035    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:03:58.334141 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.334117    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:58.334301 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:58.334279    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:03:58.354187 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.354149    2531 generic.go:356] "Generic (PLEG): container finished" podID="30f59d7fb7872ccb4c615dc2b4549c54" containerID="89822830f0e33c14bdad818047ce7cb85ae193f9852ed55501130ee1a30aeb0b" exitCode=0
Jul 27 20:03:58.354374 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.354247    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"30f59d7fb7872ccb4c615dc2b4549c54","Type":"ContainerDied","Data":"89822830f0e33c14bdad818047ce7cb85ae193f9852ed55501130ee1a30aeb0b"}
Jul 27 20:03:58.362519 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.362487    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f27hf" event={"ID":"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6","Type":"ContainerStarted","Data":"afed3099cc9eec2919a307f87ebe896b7a875f2dbc6520562a14ee271e4cd60b"}
Jul 27 20:03:58.386173 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.386074    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkvwd" event={"ID":"ad281922-6789-47b3-9d99-8920f6816fef","Type":"ContainerStarted","Data":"3722aa2dcbe78182cef6e6dc80d91ce69db807f5b568661e63f1bcb68e0a2284"}
Jul 27 20:03:58.418658 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.418611    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvkf6" event={"ID":"2732a934-0b45-4204-b0f4-677985392b1a","Type":"ContainerStarted","Data":"98f34e5fd7a14a6503887c867c9b13f6470d101ba816daca3dbc0d7b9fb5a424"}
Jul 27 20:03:58.426657 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.426618    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qd678" event={"ID":"343e6847-03ce-4a40-96ea-0e0524a2110e","Type":"ContainerStarted","Data":"3200f17797ac846357e74427bfa2f8581e925a3cb452acd95211a387d028bcfd"}
Jul 27 20:03:58.429315 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.429219    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h698x" event={"ID":"6ab5f088-3f79-4187-9206-27a21a968b13","Type":"ContainerStarted","Data":"c839f84fe75237f83c02bafc983afac81542360cb1318313f9649568bace61e9"}
Jul 27 20:03:58.438275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.437535    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"db78e8b9613587787b5e114995f93da3f61e6056b159dbe31ff0095a29351a11"}
Jul 27 20:03:58.442128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.442042    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs" event={"ID":"56d5077a-f097-4203-8c38-51963fabd5f3","Type":"ContainerStarted","Data":"6cd172bcbc8d9deb7ab6d5e8efeee250c395dfff6358a14f9f397fdc1d074faf"}
Jul 27 20:03:58.463738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.463624    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zbmqq" event={"ID":"d881ed48-6c18-454c-83e4-6dda56328c2f","Type":"ContainerStarted","Data":"ad499327fa85f802ceb3c34d353bdbbb75a4e435d7b9ca3c395b4a7b3e55c215"}
Jul 27 20:03:58.486573 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.486531    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerStarted","Data":"f7ea30f79f379b68b4cfda1dbc82e37a8af3fd0df1065860902fcc0d6369a676"}
Jul 27 20:03:58.499672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:58.499637    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"ec91069f1c74047cfed254ac36b1a990","Type":"ContainerStarted","Data":"34ed13f8d5f860177e93e5e6e36deb96a85d1b98037a6d912087b3db63168962"}
Jul 27 20:03:59.332311 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.332274    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:59.332819 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.332412    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:03:59.512420 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.512359    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"30f59d7fb7872ccb4c615dc2b4549c54","Type":"ContainerStarted","Data":"cb0905d68a1601f19be5a5a4d37771f67717f20bade311fe82f11cf877bf3228"}
Jul 27 20:03:59.528037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.526997    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" podStartSLOduration=3.526974934 podStartE2EDuration="3.526974934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:03:58.514877306 +0000 UTC m=+4.609462777" watchObservedRunningTime="2026-07-27 20:03:59.526974934 +0000 UTC m=+5.621560403"
Jul 27 20:03:59.888105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.888066    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:03:59.888324 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.888272    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:59.888405 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.888343    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:03.888319574 +0000 UTC m=+9.982905033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:03:59.989217 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.989145    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:03:59.989476 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:03:59.989226    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:03:59.989476 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989373    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:59.989476 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989437    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:04:03.989419878 +0000 UTC m=+10.084005330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:03:59.990060 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989866    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:03:59.990060 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989896    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:03:59.990060 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989909    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:03:59.990060 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:03:59.989958    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:04:03.989941398 +0000 UTC m=+10.084526847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:00.333370 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:00.333219    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:00.333882 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:00.333379    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:00.333882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:00.333814    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:00.334015 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:00.333913    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:01.332748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:01.332174    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:01.332748 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:01.332342    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:02.335007 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:02.334780    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:02.335007 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:02.334925    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:02.335578 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:02.335040    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:02.335578 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:02.335143    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:03.333002 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:03.332777    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:03.333002 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:03.332917    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:03.927930 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:03.927272    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:03.927930 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:03.927505    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:03.927930 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:03.927569    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:11.927550595 +0000 UTC m=+18.022136060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:04.028529    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:04.028599    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028787    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028804    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028833    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028848    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028852    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:04:12.028833611 +0000 UTC m=+18.123419061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:04.029224 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.028923    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:04:12.028904722 +0000 UTC m=+18.123490168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:04.333443 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:04.332853    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:04.333443 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.332988    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:04.334194 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:04.334165    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:04.334318 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:04.334291    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:05.332300 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:05.332262    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:05.332815 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:05.332397    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:06.332853 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:06.332335    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:06.332853 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:06.332500    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:06.332853 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:06.332848    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:06.333464 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:06.332963    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:07.332399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:07.332186    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:07.332399 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:07.332321    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:08.335117 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:08.335089    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:08.335540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:08.335092    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:08.335540 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:08.335249    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:08.335540 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:08.335297    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:09.332591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:09.332504    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:09.332757 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:09.332637    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:10.335355 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:10.335324    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:10.335779 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:10.335324    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:10.335779 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:10.335443    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:10.335779 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:10.335481    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:11.332484 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:11.332446    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:11.332698 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:11.332575    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:11.987944 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:11.987912    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:11.988422 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:11.988073    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:11.988422 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:11.988145    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:27.988124924 +0000 UTC m=+34.082710380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:12.089025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:12.088982    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:12.089025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:12.089030    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:12.089249 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089143    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:12.089249 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089163    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:04:12.089249 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089209    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:04:12.089249 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089222    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:12.089249 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089224    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:04:28.089201482 +0000 UTC m=+34.183786948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:12.089459 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.089267    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:04:28.089253017 +0000 UTC m=+34.183838460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:12.332892 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:12.332814    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:12.333038 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:12.332825    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:12.333038 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.332959    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:12.333038 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:12.333026    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:13.332885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:13.332842    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:13.333299 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:13.332985    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:14.335559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:14.335529    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:14.335990 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:14.335529    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:14.335990 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:14.335632    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:14.335990 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:14.335716    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:15.333062 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:15.333024    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:15.333241 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:15.333132    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:16.335624 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:16.335592    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:16.336099 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:16.335734    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:16.336099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:16.335790    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:16.336099 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:16.335903    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:17.332546 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.332215    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:17.332652 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:17.332633    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:17.557396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557359    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"22b474380d0149d78ef96b0f56d61a90f9e5a318c1bfd8f637a9a38f2f720884"}
Jul 27 20:04:17.558241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557436    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"5178b0e287af98340234a0fee14056bae8f147ed324cdb01349213e337822577"}
Jul 27 20:04:17.558241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557474    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"66be0b3004cffa6e15b15f84345392d66442cfcea26dbb23fd81303638a30fa7"}
Jul 27 20:04:17.558241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557517    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"4dbbb628adfa01219f2d90d37e2355d96de78bb3ad1ec024afc3989d6d4358bf"}
Jul 27 20:04:17.558241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557563    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"a475482a83e7fffa5cc50d920d706f1798d37e22100561c15dab1304ef59a555"}
Jul 27 20:04:17.558241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.557610    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"b88520ecd3f0bad49950260aa6f8c3008ee182d33e1fb117a0cfbbec9f399423"}
Jul 27 20:04:17.558941 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.558879    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs" event={"ID":"56d5077a-f097-4203-8c38-51963fabd5f3","Type":"ContainerStarted","Data":"20714bcbccf403ad925f1c4b9f9329f679d9f4cb3d84048643fb01b9a76e3881"}
Jul 27 20:04:17.560504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.560404    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zbmqq" event={"ID":"d881ed48-6c18-454c-83e4-6dda56328c2f","Type":"ContainerStarted","Data":"f22931e891d601a0b18d3bff532a07baa5b79bbbcc1bc291aa3db19ed4683678"}
Jul 27 20:04:17.562020 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.561978    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="91e97549b5e538ea277fb78798b74076499e115677e2c3cf333664f57a49d2f6" exitCode=0
Jul 27 20:04:17.562125 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.562060    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"91e97549b5e538ea277fb78798b74076499e115677e2c3cf333664f57a49d2f6"}
Jul 27 20:04:17.565925 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.565905    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f27hf" event={"ID":"3dd1f022-a6a1-48fa-acaf-d545c56e8ad6","Type":"ContainerStarted","Data":"fbbc190f786a78b67596b8d44f53e1636fe85b526d14e8656f36fa0cdac7e951"}
Jul 27 20:04:17.567439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.567419    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkvwd" event={"ID":"ad281922-6789-47b3-9d99-8920f6816fef","Type":"ContainerStarted","Data":"c29c8b02b06258781a434be43f1ed47d898f23cdd7f49df76bca2325f7b02909"}
Jul 27 20:04:17.568884 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.568862    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qd678" event={"ID":"343e6847-03ce-4a40-96ea-0e0524a2110e","Type":"ContainerStarted","Data":"3e810857f3524c99133acd2bd133158928a97019ffdc27b57c2956f07400080a"}
Jul 27 20:04:17.570401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.570372    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h698x" event={"ID":"6ab5f088-3f79-4187-9206-27a21a968b13","Type":"ContainerStarted","Data":"516b1b961dc5883044ea1bc738f72cebad7d303c4366df4805e4b84a3147d40f"}
Jul 27 20:04:17.581545 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.581492    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" podStartSLOduration=21.581464453 podStartE2EDuration="21.581464453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:03:59.527975298 +0000 UTC m=+5.622560772" watchObservedRunningTime="2026-07-27 20:04:17.581464453 +0000 UTC m=+23.676049896"
Jul 27 20:04:17.582581 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.582541    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zbmqq" podStartSLOduration=3.405297932 podStartE2EDuration="22.582526501s" totalImagesPullingTime="19.177228569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:17.581638976 +0000 UTC m=+23.676224454" watchObservedRunningTime="2026-07-27 20:04:17.582526501 +0000 UTC m=+23.677111968"
Jul 27 20:04:17.627125 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.627045    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rkvwd" podStartSLOduration=7.574832175 podStartE2EDuration="22.627024867s" totalImagesPullingTime="15.052192692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:17.598618048 +0000 UTC m=+23.693203513" watchObservedRunningTime="2026-07-27 20:04:17.627024867 +0000 UTC m=+23.721610333"
Jul 27 20:04:17.642681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.642634    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qd678" podStartSLOduration=3.465850774 podStartE2EDuration="22.642605682s" totalImagesPullingTime="19.176754908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:17.642584401 +0000 UTC m=+23.737169865" watchObservedRunningTime="2026-07-27 20:04:17.642605682 +0000 UTC m=+23.737191147"
Jul 27 20:04:17.662511 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.662466    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h698x" podStartSLOduration=3.440741978 podStartE2EDuration="22.662452507s" totalImagesPullingTime="19.221710529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:17.661977182 +0000 UTC m=+23.756562645" watchObservedRunningTime="2026-07-27 20:04:17.662452507 +0000 UTC m=+23.757037971"
Jul 27 20:04:17.694969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:17.694906    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f27hf" podStartSLOduration=3.5276479590000003 podStartE2EDuration="22.694887289s" totalImagesPullingTime="19.16723933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:17.694264251 +0000 UTC m=+23.788849716" watchObservedRunningTime="2026-07-27 20:04:17.694887289 +0000 UTC m=+23.789472755"
Jul 27 20:04:18.333132 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.333102    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:18.333287 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:18.333263    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:18.333372 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.333360    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:18.333503 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:18.333480    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:18.368743 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.368707    2531 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:04:18.574773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.574688    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvkf6" event={"ID":"2732a934-0b45-4204-b0f4-677985392b1a","Type":"ContainerStarted","Data":"554ad6bb2647b8a7e740ad4c814cb622ec4172127e06e1b4c3b8ae47c2e98218"}
Jul 27 20:04:18.576678 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.576643    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs" event={"ID":"56d5077a-f097-4203-8c38-51963fabd5f3","Type":"ContainerStarted","Data":"a9225cfba1ba7b06397b16d9223c3995519d2b243658d3bc64a506458717d254"}
Jul 27 20:04:18.590793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:18.590752    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dvkf6" podStartSLOduration=4.422806598 podStartE2EDuration="23.590738133s" totalImagesPullingTime="19.167931535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:18.590571737 +0000 UTC m=+24.685157201" watchObservedRunningTime="2026-07-27 20:04:18.590738133 +0000 UTC m=+24.685323598"
Jul 27 20:04:19.301928 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:19.301797    2531 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:04:18.368724602Z","UUID":"0a39944c-9af9-41d6-a0fd-66701a6f2b95","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:04:19.303711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:19.303668    2531 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:04:19.303711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:19.303714    2531 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:04:19.332755 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:19.332729    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:19.332903 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:19.332838    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:19.582829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:19.582759    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"4917e7d233d7500b9b231dc8d8ab294240f55b42baf2d3e1ac18801a101d7625"}
Jul 27 20:04:20.333237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:20.333198    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:20.333237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:20.333236    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:20.333477 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:20.333334    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:20.333703 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:20.333674    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:20.586938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:20.586854    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs" event={"ID":"56d5077a-f097-4203-8c38-51963fabd5f3","Type":"ContainerStarted","Data":"52213dacb450b97f6f848d8e12a5db5e56a130e75fc78e99d779e92bf9a0f03e"}
Jul 27 20:04:20.606099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:20.606054    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s2ggs" podStartSLOduration=4.015413461 podStartE2EDuration="25.606041525s" totalImagesPullingTime="21.590628064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:20.604949428 +0000 UTC m=+26.699534933" watchObservedRunningTime="2026-07-27 20:04:20.606041525 +0000 UTC m=+26.700626989"
Jul 27 20:04:21.116968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:21.116794    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:04:21.117409 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:21.117379    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:04:21.332834 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:21.332805    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:21.333002 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:21.332931    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:21.588913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:21.588877    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:04:21.589271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:21.589252    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkvwd"
Jul 27 20:04:22.332600 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.332414    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:22.332747 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.332428    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:22.332747 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:22.332691    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:22.332747 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:22.332739    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:22.594330 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.594255    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" event={"ID":"3bc8a0d9-73c6-418b-9e23-30a52d74e92f","Type":"ContainerStarted","Data":"5a7484d395eca62bbb31a39efe42f556ffada84e590114e529b3b93f535c342a"}
Jul 27 20:04:22.595066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.594661    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:04:22.595066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.594731    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:04:22.596625 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.596602    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="1e0d873b5ab5b32a8f55ba883a2b706ef29380c59602ab135e58dedd07a3d1e5" exitCode=0
Jul 27 20:04:22.596755 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.596693    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"1e0d873b5ab5b32a8f55ba883a2b706ef29380c59602ab135e58dedd07a3d1e5"}
Jul 27 20:04:22.610056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.610033    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:04:22.610407 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.610393    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:04:22.623682 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:22.623646    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm" podStartSLOduration=8.388259265 podStartE2EDuration="27.623632408s" totalImagesPullingTime="19.235373143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:22.622760298 +0000 UTC m=+28.717345785" watchObservedRunningTime="2026-07-27 20:04:22.623632408 +0000 UTC m=+28.718217863"
Jul 27 20:04:23.115647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.115616    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:04:23.332570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.332543    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:23.332741 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:23.332646    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:23.600603 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.600576    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="6801e747e054f07d367ab7221dd89ba04059c811f06beef080bb13e6d368ff1b" exitCode=0
Jul 27 20:04:23.600894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.600667    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"6801e747e054f07d367ab7221dd89ba04059c811f06beef080bb13e6d368ff1b"}
Jul 27 20:04:23.630841 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.630816    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fgn6x"]
Jul 27 20:04:23.630964 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.630942    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:23.631060 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:23.631034    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:23.634156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.634134    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c9krr"]
Jul 27 20:04:23.634328 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.634256    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:23.634411 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:23.634370    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:23.634989 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.634791    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dflm4"]
Jul 27 20:04:23.634989 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:23.634871    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:23.634989 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:23.634954    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:24.820624 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:24.820411    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:24.936614 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:24.936585    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:25.332356 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:25.332317    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:25.332543 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:25.332317    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:25.332543 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:25.332442    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:25.332543 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:25.332519    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:25.332543 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:25.332317    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:25.332696 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:25.332593    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:25.609118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:25.609034    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="6944fbf942c4da7f9bb41105b16803aca8a9c6ff56105735fed93e2667865182" exitCode=0
Jul 27 20:04:25.609118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:25.609067    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"6944fbf942c4da7f9bb41105b16803aca8a9c6ff56105735fed93e2667865182"}
Jul 27 20:04:27.332626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:27.332586    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:27.333291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:27.332595    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:27.333291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:27.332727    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:27.333291 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:27.332721    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:27.333291 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:27.332807    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:27.333291 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:27.332882    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:27.840788 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:27.840753    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:27.939459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:27.939427    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:28.008435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:28.008402    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:28.008626 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.008604    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:28.008699 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.008688    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:00.008665032 +0000 UTC m=+66.103250487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:04:28.109930 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:28.109845    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:28.109930 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:28.109911    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110006    2531 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110016    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110034    2531 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110046    2531 projected.go:196] Error preparing data for projected volume kube-api-access-q6n48 for pod openshift-network-diagnostics/network-check-target-fgn6x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110078    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret podName:fae3a580-5781-48b4-a8bb-8880bdcf81da nodeName:}" failed. No retries permitted until 2026-07-27 20:05:00.110059905 +0000 UTC m=+66.204645368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret") pod "global-pull-secret-syncer-dflm4" (UID: "fae3a580-5781-48b4-a8bb-8880bdcf81da") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:04:28.110149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:28.110094    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48 podName:8e00131a-aaea-4e1e-9688-f22a2f461afd nodeName:}" failed. No retries permitted until 2026-07-27 20:05:00.110087941 +0000 UTC m=+66.204673384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q6n48" (UniqueName: "kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48") pod "network-check-target-fgn6x" (UID: "8e00131a-aaea-4e1e-9688-f22a2f461afd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:04:29.333080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.333042    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:29.333080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.333056    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:29.333707 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.333042    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:29.333707 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:29.333169    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dflm4" podUID="fae3a580-5781-48b4-a8bb-8880bdcf81da"
Jul 27 20:04:29.333707 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:29.333246    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c9krr" podUID="cc8d3e6e-1331-40bf-a5cf-c461f7485f39"
Jul 27 20:04:29.333707 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:29.333301    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fgn6x" podUID="8e00131a-aaea-4e1e-9688-f22a2f461afd"
Jul 27 20:04:29.429020 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.428984    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:29.527900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.527872    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:29.620620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:29.620550    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerStarted","Data":"650a66657863fbfeadc28103ca3e1d493a1afe461f3b743bd929cb8e8c2514f4"}
Jul 27 20:04:30.625641 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.625545    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="650a66657863fbfeadc28103ca3e1d493a1afe461f3b743bd929cb8e8c2514f4" exitCode=0
Jul 27 20:04:30.625641 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.625592    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"650a66657863fbfeadc28103ca3e1d493a1afe461f3b743bd929cb8e8c2514f4"}
Jul 27 20:04:30.727842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.727820    2531 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeReady"
Jul 27 20:04:30.727988 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.727977    2531 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:04:30.774648 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.774619    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmzlx"]
Jul 27 20:04:30.777738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.777709    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:30.777848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.777797    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-krdxc"]
Jul 27 20:04:30.780851 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.780827    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:30.781087 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.781059    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:04:30.781211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.781063    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:04:30.781211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.781064    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k4bq\""
Jul 27 20:04:30.783028 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.783012    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dk9cc\""
Jul 27 20:04:30.783884 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.783864    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:04:30.783974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.783869    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:04:30.784231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.784218    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:04:30.789042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.788983    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmzlx"]
Jul 27 20:04:30.791293 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.791274    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krdxc"]
Jul 27 20:04:30.905037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.904957    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:30.932938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.932911    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkdn\" (UniqueName: \"kubernetes.io/projected/26082a5d-26b7-4e73-8131-f5446e7d9441-kube-api-access-zhkdn\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:30.933077 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.932950    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26082a5d-26b7-4e73-8131-f5446e7d9441-config-volume\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:30.933077 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.932987    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-cert\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:30.933077 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.933047    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwnq\" (UniqueName: \"kubernetes.io/projected/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-kube-api-access-qpwnq\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:30.933244 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.933092    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26082a5d-26b7-4e73-8131-f5446e7d9441-tmp-dir\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:30.933244 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:30.933133    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.005709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.005679    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:31.033821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.033798    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26082a5d-26b7-4e73-8131-f5446e7d9441-tmp-dir\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.033913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.033845    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.033913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.033887    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkdn\" (UniqueName: \"kubernetes.io/projected/26082a5d-26b7-4e73-8131-f5446e7d9441-kube-api-access-zhkdn\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.034023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.033927    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26082a5d-26b7-4e73-8131-f5446e7d9441-config-volume\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.034023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.033984    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-cert\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:31.034023 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:31.034009    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:31.034154 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:31.034068    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:31.534053425 +0000 UTC m=+37.628638868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:31.034154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.034105    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26082a5d-26b7-4e73-8131-f5446e7d9441-tmp-dir\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.034154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.034013    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwnq\" (UniqueName: \"kubernetes.io/projected/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-kube-api-access-qpwnq\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:31.034575 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.034557    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26082a5d-26b7-4e73-8131-f5446e7d9441-config-volume\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.038004 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.037885    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-cert\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:31.042656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.042635    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwnq\" (UniqueName: \"kubernetes.io/projected/f1586ff9-673a-43ad-a10c-31e8fe6aa7b2-kube-api-access-qpwnq\") pod \"ingress-canary-krdxc\" (UID: \"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2\") " pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:31.042747 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.042676    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkdn\" (UniqueName: \"kubernetes.io/projected/26082a5d-26b7-4e73-8131-f5446e7d9441-kube-api-access-zhkdn\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.097724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.097686    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krdxc"
Jul 27 20:04:31.231862 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.230746    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krdxc"]
Jul 27 20:04:31.235703 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:04:31.235673    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1586ff9_673a_43ad_a10c_31e8fe6aa7b2.slice/crio-80cefdc12b320a2b2203fd11f12201920fda556993c95717740330a8c66cefef WatchSource:0}: Error finding container 80cefdc12b320a2b2203fd11f12201920fda556993c95717740330a8c66cefef: Status 404 returned error can't find the container with id 80cefdc12b320a2b2203fd11f12201920fda556993c95717740330a8c66cefef
Jul 27 20:04:31.332935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.332904    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:04:31.332935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.332934    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:04:31.333128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.332943    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:04:31.336161 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336127    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:04:31.336161 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336146    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:04:31.336161 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336158    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5hkrg\""
Jul 27 20:04:31.336391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336226    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:04:31.336391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336147    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:04:31.336391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.336256    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nkj9h\""
Jul 27 20:04:31.538435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.538356    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:31.538574 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:31.538459    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:31.538574 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:31.538511    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:32.538496732 +0000 UTC m=+38.633082174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:31.630396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.630360    2531 generic.go:356] "Generic (PLEG): container finished" podID="1b228cf7-a5ab-410f-b58b-1bcc0008a19f" containerID="6710fa36b1faa9718fe7b2d6a86b5266aa4c960c0fc7f756318c6ceb527adca3" exitCode=0
Jul 27 20:04:31.630874 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.630430    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerDied","Data":"6710fa36b1faa9718fe7b2d6a86b5266aa4c960c0fc7f756318c6ceb527adca3"}
Jul 27 20:04:31.631769 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:31.631477    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krdxc" event={"ID":"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2","Type":"ContainerStarted","Data":"80cefdc12b320a2b2203fd11f12201920fda556993c95717740330a8c66cefef"}
Jul 27 20:04:32.546278 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:32.546077    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:32.546450 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:32.546280    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:32.546450 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:32.546363    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:34.546342054 +0000 UTC m=+40.640927498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:32.637887 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:32.637852    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" event={"ID":"1b228cf7-a5ab-410f-b58b-1bcc0008a19f","Type":"ContainerStarted","Data":"ada4d6ea1c20876eb796356ea4f75a43e2502a9eb8d2e8f18f37c0be875fde28"}
Jul 27 20:04:32.662568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:32.662488    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4dkm5" podStartSLOduration=8.597734745 podStartE2EDuration="37.662465558s" totalImagesPullingTime="29.064730813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:32.661421814 +0000 UTC m=+38.756007280" watchObservedRunningTime="2026-07-27 20:04:32.662465558 +0000 UTC m=+38.757051025"
Jul 27 20:04:32.789126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:32.789080    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:32.889618 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:32.889542    2531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log" limit=1024
Jul 27 20:04:34.561886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:34.561858    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:34.562362 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:34.561969    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:34.562362 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:34.562019    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:38.562005503 +0000 UTC m=+44.656590945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:34.643287 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:34.643257    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krdxc" event={"ID":"f1586ff9-673a-43ad-a10c-31e8fe6aa7b2","Type":"ContainerStarted","Data":"1b15b8d0d2799a61535cb4ca04acf688746db816e370fb50665b0c1ea05750c3"}
Jul 27 20:04:34.659793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:34.659723    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-krdxc" podStartSLOduration=1.72801702 podStartE2EDuration="4.659706999s" totalImagesPullingTime="2.931689979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:04:34.658934626 +0000 UTC m=+40.753520094" watchObservedRunningTime="2026-07-27 20:04:34.659706999 +0000 UTC m=+40.754292466"
Jul 27 20:04:38.590276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:38.590236    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:38.590649 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:38.590387    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:38.590649 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:38.590449    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:04:46.590433704 +0000 UTC m=+52.685019146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:46.645102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:46.645061    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:04:46.645530 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:46.645240    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:04:46.645530 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:04:46.645306    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:02.645289469 +0000 UTC m=+68.739874916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:04:54.619239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:04:54.619212    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6wqxm"
Jul 27 20:05:00.032687 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.032629    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:05:00.035510 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.035488    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:05:00.042852 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:00.042825    2531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 20:05:00.042927 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:00.042915    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs podName:cc8d3e6e-1331-40bf-a5cf-c461f7485f39 nodeName:}" failed. No retries permitted until 2026-07-27 20:06:04.042899233 +0000 UTC m=+130.137484680 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs") pod "network-metrics-daemon-c9krr" (UID: "cc8d3e6e-1331-40bf-a5cf-c461f7485f39") : secret "metrics-daemon-secret" not found
Jul 27 20:05:00.134014 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.133965    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:05:00.134118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.134031    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:05:00.136762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.136741    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:05:00.136811 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.136759    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:05:00.147263 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.147224    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:05:00.148475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.148439    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fae3a580-5781-48b4-a8bb-8880bdcf81da-original-pull-secret\") pod \"global-pull-secret-syncer-dflm4\" (UID: \"fae3a580-5781-48b4-a8bb-8880bdcf81da\") " pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:05:00.157558 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.157530    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6n48\" (UniqueName: \"kubernetes.io/projected/8e00131a-aaea-4e1e-9688-f22a2f461afd-kube-api-access-q6n48\") pod \"network-check-target-fgn6x\" (UID: \"8e00131a-aaea-4e1e-9688-f22a2f461afd\") " pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:05:00.442841 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.442737    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dflm4"
Jul 27 20:05:00.456023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.455998    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nkj9h\""
Jul 27 20:05:00.464051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.464027    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:05:00.595894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.595865    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dflm4"]
Jul 27 20:05:00.599751 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:00.599726    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae3a580_5781_48b4_a8bb_8880bdcf81da.slice/crio-0512252dab413c42b4dca35737a71fdfa4833aafb37c2e520e86163ce229792f WatchSource:0}: Error finding container 0512252dab413c42b4dca35737a71fdfa4833aafb37c2e520e86163ce229792f: Status 404 returned error can't find the container with id 0512252dab413c42b4dca35737a71fdfa4833aafb37c2e520e86163ce229792f
Jul 27 20:05:00.613727 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.613698    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fgn6x"]
Jul 27 20:05:00.616608 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:00.616582    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e00131a_aaea_4e1e_9688_f22a2f461afd.slice/crio-d14ac329cf1a5437a800deb6bf4020f658d21d8353f4edb0c185c8271e8e3136 WatchSource:0}: Error finding container d14ac329cf1a5437a800deb6bf4020f658d21d8353f4edb0c185c8271e8e3136: Status 404 returned error can't find the container with id d14ac329cf1a5437a800deb6bf4020f658d21d8353f4edb0c185c8271e8e3136
Jul 27 20:05:00.702239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.702147    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fgn6x" event={"ID":"8e00131a-aaea-4e1e-9688-f22a2f461afd","Type":"ContainerStarted","Data":"d14ac329cf1a5437a800deb6bf4020f658d21d8353f4edb0c185c8271e8e3136"}
Jul 27 20:05:00.703117 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:00.703091    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dflm4" event={"ID":"fae3a580-5781-48b4-a8bb-8880bdcf81da","Type":"ContainerStarted","Data":"0512252dab413c42b4dca35737a71fdfa4833aafb37c2e520e86163ce229792f"}
Jul 27 20:05:02.650444 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:02.650402    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:05:02.650835 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:02.650595    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:05:02.650835 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:02.650680    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:34.650659098 +0000 UTC m=+100.745244546 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:05:04.715727 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:04.715688    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fgn6x" event={"ID":"8e00131a-aaea-4e1e-9688-f22a2f461afd","Type":"ContainerStarted","Data":"e1c4b65cfc88821c91ad748ff28ff5c38beb18ac9d469467121a5667e0288cf2"}
Jul 27 20:05:04.716152 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:04.715918    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:05:04.735555 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:04.735482    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fgn6x" podStartSLOduration=66.485225021 podStartE2EDuration="1m9.735464849s" totalImagesPullingTime="3.250239828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:04.73468097 +0000 UTC m=+70.829266435" watchObservedRunningTime="2026-07-27 20:05:04.735464849 +0000 UTC m=+70.830050292"
Jul 27 20:05:05.719338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:05.719247    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dflm4" event={"ID":"fae3a580-5781-48b4-a8bb-8880bdcf81da","Type":"ContainerStarted","Data":"c231ce1714f9c282ae55cd3fcb30606c926dc29bca23dc95972ef53a4f9ccdf8"}
Jul 27 20:05:05.739331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:05.739268    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dflm4" podStartSLOduration=65.914769969 podStartE2EDuration="1m10.739248238s" totalImagesPullingTime="4.824478269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:05.738433991 +0000 UTC m=+71.833019456" watchObservedRunningTime="2026-07-27 20:05:05.739248238 +0000 UTC m=+71.833833703"
Jul 27 20:05:33.214132 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.214096    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"]
Jul 27 20:05:33.218571 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.218549    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.220901 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.220880    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 20:05:33.221010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.220886    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 20:05:33.221909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.221889    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2bq9t\""
Jul 27 20:05:33.222794 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.222326    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:05:33.222794 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.222427    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 20:05:33.228491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.228445    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"]
Jul 27 20:05:33.253293 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.253260    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5209a37f-3e8b-4166-a6a7-df67e9a2db38-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.253293 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.253293    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5209a37f-3e8b-4166-a6a7-df67e9a2db38-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.253448 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.253339    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcbk\" (UniqueName: \"kubernetes.io/projected/5209a37f-3e8b-4166-a6a7-df67e9a2db38-kube-api-access-mzcbk\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.319283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.319229    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"]
Jul 27 20:05:33.321945 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.321930    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j7tjq"]
Jul 27 20:05:33.322092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.322071    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.324782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.324762    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.324996 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.324971    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s82sn\""
Jul 27 20:05:33.326263 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326244    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:05:33.326351 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326254    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:05:33.326412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326288    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 20:05:33.326563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326543    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 20:05:33.326765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326751    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:05:33.326837 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.326767    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 20:05:33.327303 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.327288    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bdv8g\""
Jul 27 20:05:33.327823 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.327807    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 20:05:33.328637 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.328397    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 20:05:33.333196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.333162    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 20:05:33.334528 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.334506    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"]
Jul 27 20:05:33.335482 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.335459    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j7tjq"]
Jul 27 20:05:33.337095 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.337073    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f24xp"]
Jul 27 20:05:33.339821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.339805    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.342026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.342005    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:05:33.342026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.342020    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 27 20:05:33.342166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.342024    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:05:33.342166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.342041    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 27 20:05:33.342166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.342012    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jzbd7\""
Jul 27 20:05:33.346937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.346918    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 27 20:05:33.347494 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.347474    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f24xp"]
Jul 27 20:05:33.354336 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354318    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlcm\" (UniqueName: \"kubernetes.io/projected/f676e8c6-2bdd-4286-96ec-79acb3a93e90-kube-api-access-fzlcm\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.354433 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354355    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5209a37f-3e8b-4166-a6a7-df67e9a2db38-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.354433 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354380    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5209a37f-3e8b-4166-a6a7-df67e9a2db38-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.354540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354440    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f676e8c6-2bdd-4286-96ec-79acb3a93e90-serving-cert\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.354540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354478    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.354540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354515    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cm5\" (UniqueName: \"kubernetes.io/projected/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-kube-api-access-q7cm5\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.354672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354556    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcbk\" (UniqueName: \"kubernetes.io/projected/5209a37f-3e8b-4166-a6a7-df67e9a2db38-kube-api-access-mzcbk\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.354672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354580    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.354672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354617    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-config\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.354814 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354693    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-trusted-ca\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.354916 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.354901    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5209a37f-3e8b-4166-a6a7-df67e9a2db38-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.356526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.356512    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5209a37f-3e8b-4166-a6a7-df67e9a2db38-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.363445 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.363423    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcbk\" (UniqueName: \"kubernetes.io/projected/5209a37f-3e8b-4166-a6a7-df67e9a2db38-kube-api-access-mzcbk\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv\" (UID: \"5209a37f-3e8b-4166-a6a7-df67e9a2db38\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.455491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455391    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.455491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455436    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-service-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.455681 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:33.455534    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:33.455681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455553    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-snapshots\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.455681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455586    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq6n\" (UniqueName: \"kubernetes.io/projected/03a10cc3-eb6e-4725-bc99-b490924fa229-kube-api-access-qkq6n\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.455681 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:33.455624    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:33.955594193 +0000 UTC m=+100.050179637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:33.455886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455694    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cm5\" (UniqueName: \"kubernetes.io/projected/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-kube-api-access-q7cm5\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.455886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455739    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.455886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455756    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-config\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455890    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-trusted-ca\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455918    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlcm\" (UniqueName: \"kubernetes.io/projected/f676e8c6-2bdd-4286-96ec-79acb3a93e90-kube-api-access-fzlcm\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455941    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455962    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-tmp\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.455991    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03a10cc3-eb6e-4725-bc99-b490924fa229-serving-cert\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.456022 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.456019    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f676e8c6-2bdd-4286-96ec-79acb3a93e90-serving-cert\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.456550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.456517    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-config\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.456698 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.456610    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.456853 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.456833    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f676e8c6-2bdd-4286-96ec-79acb3a93e90-trusted-ca\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.458274 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.458255    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f676e8c6-2bdd-4286-96ec-79acb3a93e90-serving-cert\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.470536 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.470514    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlcm\" (UniqueName: \"kubernetes.io/projected/f676e8c6-2bdd-4286-96ec-79acb3a93e90-kube-api-access-fzlcm\") pod \"console-operator-584dcb6b54-j7tjq\" (UID: \"f676e8c6-2bdd-4286-96ec-79acb3a93e90\") " pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.470626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.470605    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cm5\" (UniqueName: \"kubernetes.io/projected/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-kube-api-access-q7cm5\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.528424 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.528389    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"
Jul 27 20:05:33.556828 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556799    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.556951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556834    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-tmp\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.556951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556851    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03a10cc3-eb6e-4725-bc99-b490924fa229-serving-cert\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.556951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556885    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-service-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.556951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556911    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-snapshots\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.556951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.556935    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq6n\" (UniqueName: \"kubernetes.io/projected/03a10cc3-eb6e-4725-bc99-b490924fa229-kube-api-access-qkq6n\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.557591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.557553    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-service-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.557681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.557617    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-snapshots\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.557871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.557853    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a10cc3-eb6e-4725-bc99-b490924fa229-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.557985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.557966    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03a10cc3-eb6e-4725-bc99-b490924fa229-tmp\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.559152 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.559132    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/03a10cc3-eb6e-4725-bc99-b490924fa229-serving-cert\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.566141 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.566117    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq6n\" (UniqueName: \"kubernetes.io/projected/03a10cc3-eb6e-4725-bc99-b490924fa229-kube-api-access-qkq6n\") pod \"insights-operator-5885fdb759-f24xp\" (UID: \"03a10cc3-eb6e-4725-bc99-b490924fa229\") " pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.638751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.638719    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:33.648203 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.648108    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-f24xp"
Jul 27 20:05:33.649980 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.649960    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv"]
Jul 27 20:05:33.652246 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:33.652224    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5209a37f_3e8b_4166_a6a7_df67e9a2db38.slice/crio-8fe4baa8587df39df2ed1f9d40a211b378bbd3018d2ce5fa05b4fd8654ffc57c WatchSource:0}: Error finding container 8fe4baa8587df39df2ed1f9d40a211b378bbd3018d2ce5fa05b4fd8654ffc57c: Status 404 returned error can't find the container with id 8fe4baa8587df39df2ed1f9d40a211b378bbd3018d2ce5fa05b4fd8654ffc57c
Jul 27 20:05:33.768759 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.768732    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-j7tjq"]
Jul 27 20:05:33.771319 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:33.771293    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf676e8c6_2bdd_4286_96ec_79acb3a93e90.slice/crio-a82194e525cd3e8b906ac16dfb6c69a69f429e2de20a061678099e826ae984a1 WatchSource:0}: Error finding container a82194e525cd3e8b906ac16dfb6c69a69f429e2de20a061678099e826ae984a1: Status 404 returned error can't find the container with id a82194e525cd3e8b906ac16dfb6c69a69f429e2de20a061678099e826ae984a1
Jul 27 20:05:33.786428 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.786401    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f24xp"]
Jul 27 20:05:33.788671 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.788641    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv" event={"ID":"5209a37f-3e8b-4166-a6a7-df67e9a2db38","Type":"ContainerStarted","Data":"8fe4baa8587df39df2ed1f9d40a211b378bbd3018d2ce5fa05b4fd8654ffc57c"}
Jul 27 20:05:33.788941 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:33.788919    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a10cc3_eb6e_4725_bc99_b490924fa229.slice/crio-e0fc9de334cce1f73c910535212c98c82c00b0608ee3acf7620a1f0ee4a666b2 WatchSource:0}: Error finding container e0fc9de334cce1f73c910535212c98c82c00b0608ee3acf7620a1f0ee4a666b2: Status 404 returned error can't find the container with id e0fc9de334cce1f73c910535212c98c82c00b0608ee3acf7620a1f0ee4a666b2
Jul 27 20:05:33.789748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.789728    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" event={"ID":"f676e8c6-2bdd-4286-96ec-79acb3a93e90","Type":"ContainerStarted","Data":"a82194e525cd3e8b906ac16dfb6c69a69f429e2de20a061678099e826ae984a1"}
Jul 27 20:05:33.960382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:33.960348    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:33.960549 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:33.960458    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:33.960549 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:33.960520    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:34.960505279 +0000 UTC m=+101.055090722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:34.666053 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:34.665414    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:05:34.666053 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:34.665606    2531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:05:34.666053 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:34.665668    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls podName:26082a5d-26b7-4e73-8131-f5446e7d9441 nodeName:}" failed. No retries permitted until 2026-07-27 20:06:38.665649164 +0000 UTC m=+164.760234614 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls") pod "dns-default-xmzlx" (UID: "26082a5d-26b7-4e73-8131-f5446e7d9441") : secret "dns-default-metrics-tls" not found
Jul 27 20:05:34.794973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:34.794894    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f24xp" event={"ID":"03a10cc3-eb6e-4725-bc99-b490924fa229","Type":"ContainerStarted","Data":"e0fc9de334cce1f73c910535212c98c82c00b0608ee3acf7620a1f0ee4a666b2"}
Jul 27 20:05:34.968371 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:34.968275    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:34.968533 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:34.968456    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:34.968533 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:34.968530    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:36.968511135 +0000 UTC m=+103.063096581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:35.722435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:35.722398    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fgn6x"
Jul 27 20:05:36.984482 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:36.984427    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:36.984930 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:36.984581    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:36.984930 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:36.984643    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:40.984627006 +0000 UTC m=+107.079212455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:37.805324 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.805290    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/0.log"
Jul 27 20:05:37.805503 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.805339    2531 generic.go:356] "Generic (PLEG): container finished" podID="f676e8c6-2bdd-4286-96ec-79acb3a93e90" containerID="366b55a6dd58d46a9afa243ec677c71a8c39b1b2de8dc94013d0d6a1cf5fff77" exitCode=255
Jul 27 20:05:37.805503 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.805433    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" event={"ID":"f676e8c6-2bdd-4286-96ec-79acb3a93e90","Type":"ContainerDied","Data":"366b55a6dd58d46a9afa243ec677c71a8c39b1b2de8dc94013d0d6a1cf5fff77"}
Jul 27 20:05:37.805666 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.805650    2531 scope.go:117] "RemoveContainer" containerID="366b55a6dd58d46a9afa243ec677c71a8c39b1b2de8dc94013d0d6a1cf5fff77"
Jul 27 20:05:37.806919 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.806892    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv" event={"ID":"5209a37f-3e8b-4166-a6a7-df67e9a2db38","Type":"ContainerStarted","Data":"de83cab6745c3b3ea7bdd062ea44dadaa1aae5523d0152796af2ca49c99cdf15"}
Jul 27 20:05:37.808276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.808250    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f24xp" event={"ID":"03a10cc3-eb6e-4725-bc99-b490924fa229","Type":"ContainerStarted","Data":"c1cc8da62077870a74ebc2bd98ff0de79d4ea2cf21dd9132ba210bf14276d6e4"}
Jul 27 20:05:37.838435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.838389    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv" podStartSLOduration=1.029677369 podStartE2EDuration="4.838370734s" totalImagesPullingTime="3.808693365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:37.83719791 +0000 UTC m=+103.931783376" watchObservedRunningTime="2026-07-27 20:05:37.838370734 +0000 UTC m=+103.932956200"
Jul 27 20:05:37.853017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:37.852160    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-f24xp" podStartSLOduration=1.172224068 podStartE2EDuration="4.852142378s" totalImagesPullingTime="3.67991831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:37.85204735 +0000 UTC m=+103.946632815" watchObservedRunningTime="2026-07-27 20:05:37.852142378 +0000 UTC m=+103.946727842"
Jul 27 20:05:38.737940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.737909    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7cpd7"]
Jul 27 20:05:38.740646 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.740630    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.749158 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.749093    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2fdzc\""
Jul 27 20:05:38.749158 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.749116    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:05:38.749371 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.749226    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:05:38.757985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.757935    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cpd7"]
Jul 27 20:05:38.797218 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.797190    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cfab18-fb28-417f-b0e9-7eee826ec06f-data-volume\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.797364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.797245    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wgz\" (UniqueName: \"kubernetes.io/projected/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-api-access-68wgz\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.797364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.797319    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9cfab18-fb28-417f-b0e9-7eee826ec06f-crio-socket\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.797476 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.797383    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.797476 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.797404    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.813232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813201    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:05:38.813613 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813598    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/0.log"
Jul 27 20:05:38.813673 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813632    2531 generic.go:356] "Generic (PLEG): container finished" podID="f676e8c6-2bdd-4286-96ec-79acb3a93e90" containerID="108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c" exitCode=255
Jul 27 20:05:38.813728 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813666    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" event={"ID":"f676e8c6-2bdd-4286-96ec-79acb3a93e90","Type":"ContainerDied","Data":"108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c"}
Jul 27 20:05:38.813795 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813733    2531 scope.go:117] "RemoveContainer" containerID="366b55a6dd58d46a9afa243ec677c71a8c39b1b2de8dc94013d0d6a1cf5fff77"
Jul 27 20:05:38.813969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.813947    2531 scope.go:117] "RemoveContainer" containerID="108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c"
Jul 27 20:05:38.814145 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:38.814121    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j7tjq_openshift-console-operator(f676e8c6-2bdd-4286-96ec-79acb3a93e90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" podUID="f676e8c6-2bdd-4286-96ec-79acb3a93e90"
Jul 27 20:05:38.898092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898056    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898091    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898348 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:38.898221    2531 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:05:38.898348 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:38.898297    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:39.398279281 +0000 UTC m=+105.492864743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : configmap "kube-rbac-proxy" not found
Jul 27 20:05:38.898348 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:38.898232    2531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:38.898509 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:38.898373    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:39.398357353 +0000 UTC m=+105.492942803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:38.898509 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898419    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cfab18-fb28-417f-b0e9-7eee826ec06f-data-volume\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898509 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898505    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wgz\" (UniqueName: \"kubernetes.io/projected/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-api-access-68wgz\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898555    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9cfab18-fb28-417f-b0e9-7eee826ec06f-crio-socket\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898719 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898697    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9cfab18-fb28-417f-b0e9-7eee826ec06f-crio-socket\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.898761 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.898747    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cfab18-fb28-417f-b0e9-7eee826ec06f-data-volume\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:38.911298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:38.911267    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wgz\" (UniqueName: \"kubernetes.io/projected/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-api-access-68wgz\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:39.402041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.402000    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:39.402041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.402044    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:39.402278 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:39.402164    2531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:39.402278 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:39.402223    2531 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:05:39.402278 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:39.402250    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:40.402232914 +0000 UTC m=+106.496818360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:39.402376 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:39.402290    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:40.402276192 +0000 UTC m=+106.496861634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : configmap "kube-rbac-proxy" not found
Jul 27 20:05:39.817616 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.817589    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:05:39.817980 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.817914    2531 scope.go:117] "RemoveContainer" containerID="108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c"
Jul 27 20:05:39.818072 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:39.818052    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j7tjq_openshift-console-operator(f676e8c6-2bdd-4286-96ec-79acb3a93e90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" podUID="f676e8c6-2bdd-4286-96ec-79acb3a93e90"
Jul 27 20:05:39.994734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.994700    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rdjqh"]
Jul 27 20:05:39.998664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:39.998647    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.000875 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.000854    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 20:05:40.001004 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.000873    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 20:05:40.001151 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.001135    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vdwmv\""
Jul 27 20:05:40.001224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.001148    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 20:05:40.001224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.001144    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 20:05:40.006503 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.006471    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rdjqh"]
Jul 27 20:05:40.107572 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.107475    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac58f39d-f655-4244-81f2-4afd041925da-signing-cabundle\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.107572 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.107533    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac58f39d-f655-4244-81f2-4afd041925da-signing-key\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.107755 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.107588    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp786\" (UniqueName: \"kubernetes.io/projected/ac58f39d-f655-4244-81f2-4afd041925da-kube-api-access-vp786\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.208832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.208795    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac58f39d-f655-4244-81f2-4afd041925da-signing-cabundle\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.208992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.208852    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac58f39d-f655-4244-81f2-4afd041925da-signing-key\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.208992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.208870    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp786\" (UniqueName: \"kubernetes.io/projected/ac58f39d-f655-4244-81f2-4afd041925da-kube-api-access-vp786\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.209451 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.209427    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ac58f39d-f655-4244-81f2-4afd041925da-signing-cabundle\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.211258 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.211238    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ac58f39d-f655-4244-81f2-4afd041925da-signing-key\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.217034 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.217011    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp786\" (UniqueName: \"kubernetes.io/projected/ac58f39d-f655-4244-81f2-4afd041925da-kube-api-access-vp786\") pod \"service-ca-658d7b7c78-rdjqh\" (UID: \"ac58f39d-f655-4244-81f2-4afd041925da\") " pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.308309 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.308275    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh"
Jul 27 20:05:40.409904 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.409867    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:40.409904 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.409906    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:40.410111 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:40.410017    2531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:40.410111 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:40.410059    2531 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:05:40.410111 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:40.410088    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:42.410067873 +0000 UTC m=+108.504653320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:40.410250 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:40.410122    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:42.410103659 +0000 UTC m=+108.504689119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : configmap "kube-rbac-proxy" not found
Jul 27 20:05:40.419105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.419080    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"]
Jul 27 20:05:40.422918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.422899    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"
Jul 27 20:05:40.425167 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.425147    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 20:05:40.425519 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.425499    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 20:05:40.425588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.425538    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vr7fj\""
Jul 27 20:05:40.430507 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.430485    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"]
Jul 27 20:05:40.434505 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.434486    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rdjqh"]
Jul 27 20:05:40.437254 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:40.437233    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac58f39d_f655_4244_81f2_4afd041925da.slice/crio-32a6dfcc87c167bb7c6a5f91d1009f57922fb04b964372cb000e1fef0f6ad9c3 WatchSource:0}: Error finding container 32a6dfcc87c167bb7c6a5f91d1009f57922fb04b964372cb000e1fef0f6ad9c3: Status 404 returned error can't find the container with id 32a6dfcc87c167bb7c6a5f91d1009f57922fb04b964372cb000e1fef0f6ad9c3
Jul 27 20:05:40.511457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.511420    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7zn\" (UniqueName: \"kubernetes.io/projected/9a528fdd-b5cc-4f5d-a223-991969b3a83c-kube-api-access-mn7zn\") pod \"migrator-6865c9967-6p96k\" (UID: \"9a528fdd-b5cc-4f5d-a223-991969b3a83c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"
Jul 27 20:05:40.611819 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.611785    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7zn\" (UniqueName: \"kubernetes.io/projected/9a528fdd-b5cc-4f5d-a223-991969b3a83c-kube-api-access-mn7zn\") pod \"migrator-6865c9967-6p96k\" (UID: \"9a528fdd-b5cc-4f5d-a223-991969b3a83c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"
Jul 27 20:05:40.634969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.634918    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7zn\" (UniqueName: \"kubernetes.io/projected/9a528fdd-b5cc-4f5d-a223-991969b3a83c-kube-api-access-mn7zn\") pod \"migrator-6865c9967-6p96k\" (UID: \"9a528fdd-b5cc-4f5d-a223-991969b3a83c\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"
Jul 27 20:05:40.720001 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.719981    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f27hf_3dd1f022-a6a1-48fa-acaf-d545c56e8ad6/dns-node-resolver/0.log"
Jul 27 20:05:40.732112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.732078    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"
Jul 27 20:05:40.822804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.822768    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh" event={"ID":"ac58f39d-f655-4244-81f2-4afd041925da","Type":"ContainerStarted","Data":"32a6dfcc87c167bb7c6a5f91d1009f57922fb04b964372cb000e1fef0f6ad9c3"}
Jul 27 20:05:40.850496 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:40.850473    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k"]
Jul 27 20:05:40.852836 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:40.852810    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a528fdd_b5cc_4f5d_a223_991969b3a83c.slice/crio-22d4c65b7ba245ec08f4b2ea0b97fbae78539cfb134ccbc6334d57798d58b32e WatchSource:0}: Error finding container 22d4c65b7ba245ec08f4b2ea0b97fbae78539cfb134ccbc6334d57798d58b32e: Status 404 returned error can't find the container with id 22d4c65b7ba245ec08f4b2ea0b97fbae78539cfb134ccbc6334d57798d58b32e
Jul 27 20:05:41.015367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:41.015296    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:41.015494 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:41.015441    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:41.015534 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:41.015503    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:05:49.015488846 +0000 UTC m=+115.110074289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:41.718459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:41.718430    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qd678_343e6847-03ce-4a40-96ea-0e0524a2110e/node-ca/0.log"
Jul 27 20:05:41.827085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:41.827047    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k" event={"ID":"9a528fdd-b5cc-4f5d-a223-991969b3a83c","Type":"ContainerStarted","Data":"22d4c65b7ba245ec08f4b2ea0b97fbae78539cfb134ccbc6334d57798d58b32e"}
Jul 27 20:05:42.427547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.427505    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:42.427547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.427551    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:42.427781 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:42.427667    2531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:42.427781 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:42.427701    2531 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:05:42.427781 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:42.427743    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:46.427723024 +0000 UTC m=+112.522308479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:42.427781 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:42.427766    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:46.427755629 +0000 UTC m=+112.522341087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : configmap "kube-rbac-proxy" not found
Jul 27 20:05:42.523397 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.523346    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-krdxc_f1586ff9-673a-43ad-a10c-31e8fe6aa7b2/serve-healthcheck-canary/0.log"
Jul 27 20:05:42.831366 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.831320    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh" event={"ID":"ac58f39d-f655-4244-81f2-4afd041925da","Type":"ContainerStarted","Data":"f5b72809c7c28332bd0200508dfd3245a2c95f62bcf4a3ce0a5b361afb3de534"}
Jul 27 20:05:42.832975 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.832948    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k" event={"ID":"9a528fdd-b5cc-4f5d-a223-991969b3a83c","Type":"ContainerStarted","Data":"40a3a9432d0ca22aef2fc3efa444e1efb6ce2a7f6d885df4d9ded89c0d8c9cb4"}
Jul 27 20:05:42.833086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.832990    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k" event={"ID":"9a528fdd-b5cc-4f5d-a223-991969b3a83c","Type":"ContainerStarted","Data":"e18f3566d87051cd5c5b6caf025ffa5062397ae5c9a9d78448a37f8bf81791d3"}
Jul 27 20:05:42.847721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.847676    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-rdjqh" podStartSLOduration=1.706765726 podStartE2EDuration="3.847662039s" totalImagesPullingTime="2.140896313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:42.846560465 +0000 UTC m=+108.941145930" watchObservedRunningTime="2026-07-27 20:05:42.847662039 +0000 UTC m=+108.942247503"
Jul 27 20:05:42.862164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:42.862118    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6p96k" podStartSLOduration=1.142700702 podStartE2EDuration="2.862103517s" totalImagesPullingTime="1.719402815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:42.860954049 +0000 UTC m=+108.955539513" watchObservedRunningTime="2026-07-27 20:05:42.862103517 +0000 UTC m=+108.956688979"
Jul 27 20:05:43.639490 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:43.639456    2531 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:43.639685 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:43.639519    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:43.639935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:43.639922    2531 scope.go:117] "RemoveContainer" containerID="108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c"
Jul 27 20:05:43.640136 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:43.640094    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-j7tjq_openshift-console-operator(f676e8c6-2bdd-4286-96ec-79acb3a93e90)\"" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" podUID="f676e8c6-2bdd-4286-96ec-79acb3a93e90"
Jul 27 20:05:46.461490 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:46.461449    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:46.461490 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:46.461493    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:46.461938 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:46.461555    2531 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:05:46.461938 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:46.461588    2531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:46.461938 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:46.461606    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:54.461590627 +0000 UTC m=+120.556176069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : configmap "kube-rbac-proxy" not found
Jul 27 20:05:46.461938 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:46.461638    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls podName:c9cfab18-fb28-417f-b0e9-7eee826ec06f nodeName:}" failed. No retries permitted until 2026-07-27 20:05:54.461623871 +0000 UTC m=+120.556209317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7cpd7" (UID: "c9cfab18-fb28-417f-b0e9-7eee826ec06f") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:05:49.081765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:49.081730    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:05:49.082152 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:49.081882    2531 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:49.082152 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:05:49.081947    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls podName:ff75a7d2-5329-46e6-86bd-2a9a017a1d73 nodeName:}" failed. No retries permitted until 2026-07-27 20:06:05.081932969 +0000 UTC m=+131.176518411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pkxt5" (UID: "ff75a7d2-5329-46e6-86bd-2a9a017a1d73") : secret "cluster-monitoring-operator-tls" not found
Jul 27 20:05:54.519475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.519424    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:54.519475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.519473    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:54.519933 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.519915    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9cfab18-fb28-417f-b0e9-7eee826ec06f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:54.521568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.521541    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9cfab18-fb28-417f-b0e9-7eee826ec06f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7cpd7\" (UID: \"c9cfab18-fb28-417f-b0e9-7eee826ec06f\") " pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:54.653384 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.653354    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2fdzc\""
Jul 27 20:05:54.662237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.662218    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7cpd7"
Jul 27 20:05:54.789500 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.789358    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7cpd7"]
Jul 27 20:05:54.792028 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:05:54.792000    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cfab18_fb28_417f_b0e9_7eee826ec06f.slice/crio-aa32b5d11aa26fd239dceae983b8af7bda7d4cd2b3d37f04178d6b9866c1d5a2 WatchSource:0}: Error finding container aa32b5d11aa26fd239dceae983b8af7bda7d4cd2b3d37f04178d6b9866c1d5a2: Status 404 returned error can't find the container with id aa32b5d11aa26fd239dceae983b8af7bda7d4cd2b3d37f04178d6b9866c1d5a2
Jul 27 20:05:54.872284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:54.872252    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cpd7" event={"ID":"c9cfab18-fb28-417f-b0e9-7eee826ec06f","Type":"ContainerStarted","Data":"aa32b5d11aa26fd239dceae983b8af7bda7d4cd2b3d37f04178d6b9866c1d5a2"}
Jul 27 20:05:56.332821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:56.332793    2531 scope.go:117] "RemoveContainer" containerID="108ba7f997aad4cbf995e12b936def8b91812c72d4321acb1bb69d986b58a58c"
Jul 27 20:05:56.880972 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:56.880945    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:05:56.881129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:56.881009    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" event={"ID":"f676e8c6-2bdd-4286-96ec-79acb3a93e90","Type":"ContainerStarted","Data":"380428b15b1e6b8b116275de720c395f2779e71e18e3cf4f4e04368c7554ea78"}
Jul 27 20:05:56.881412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:56.881391    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:56.900420 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:56.900372    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq" podStartSLOduration=20.208978471 podStartE2EDuration="23.900357665s" totalImagesPullingTime="3.691379194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:05:56.899673899 +0000 UTC m=+122.994259362" watchObservedRunningTime="2026-07-27 20:05:56.900357665 +0000 UTC m=+122.994943168"
Jul 27 20:05:57.490302 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:57.490270    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-j7tjq"
Jul 27 20:05:59.891741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:05:59.891667    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cpd7" event={"ID":"c9cfab18-fb28-417f-b0e9-7eee826ec06f","Type":"ContainerStarted","Data":"a9bec80b209fe42762e21c5011581bcfc7ef11dd47cee28ced1df93e44a76a6b"}
Jul 27 20:06:00.446137 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.446105    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d459c79d7-c5g59"]
Jul 27 20:06:00.449275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.449256    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.457257 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.457000    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 20:06:00.457257 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.457076    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 20:06:00.458034 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.458011    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bsfqh\""
Jul 27 20:06:00.458140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.458014    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 20:06:00.462282 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.462264    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 20:06:00.473634 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.473578    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d459c79d7-c5g59"]
Jul 27 20:06:00.570438 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570314    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-trusted-ca\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570631 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570454    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp56p\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-kube-api-access-pp56p\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570631 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570506    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-tls\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570531    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-bound-sa-token\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570719    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50cc35a-ab6b-4924-a712-2f44b4413c9d-ca-trust-extracted\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570787    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-image-registry-private-configuration\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570817    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-certificates\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.570949 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.570902    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-installation-pull-secrets\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671729 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671689    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-image-registry-private-configuration\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671729 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671733    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-certificates\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671799    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-installation-pull-secrets\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671834    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-trusted-ca\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671858    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp56p\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-kube-api-access-pp56p\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671889    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-tls\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671910    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-bound-sa-token\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.671983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.671934    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50cc35a-ab6b-4924-a712-2f44b4413c9d-ca-trust-extracted\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.672325 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.672289    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50cc35a-ab6b-4924-a712-2f44b4413c9d-ca-trust-extracted\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.673157 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.673101    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-certificates\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.673766 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.673737    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50cc35a-ab6b-4924-a712-2f44b4413c9d-trusted-ca\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.675196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.675135    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-registry-tls\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.676033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.676011    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-image-registry-private-configuration\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.676559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.676537    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50cc35a-ab6b-4924-a712-2f44b4413c9d-installation-pull-secrets\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.686816 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.686762    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp56p\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-kube-api-access-pp56p\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.688929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.688896    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50cc35a-ab6b-4924-a712-2f44b4413c9d-bound-sa-token\") pod \"image-registry-7d459c79d7-c5g59\" (UID: \"a50cc35a-ab6b-4924-a712-2f44b4413c9d\") " pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.759951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.759883    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:00.925376 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:00.925349    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d459c79d7-c5g59"]
Jul 27 20:06:00.928019 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:00.927992    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50cc35a_ab6b_4924_a712_2f44b4413c9d.slice/crio-a8cc55ee5fefe3035e5cbcfb1691c9b10ccefe46de2629cd7345e576a534e838 WatchSource:0}: Error finding container a8cc55ee5fefe3035e5cbcfb1691c9b10ccefe46de2629cd7345e576a534e838: Status 404 returned error can't find the container with id a8cc55ee5fefe3035e5cbcfb1691c9b10ccefe46de2629cd7345e576a534e838
Jul 27 20:06:01.901125 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:01.901081    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59" event={"ID":"a50cc35a-ab6b-4924-a712-2f44b4413c9d","Type":"ContainerStarted","Data":"21057d113b911e10308ddb9d8dc75452dc3a996628ea6beed1daf3498d3228d4"}
Jul 27 20:06:01.901327 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:01.901200    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59" event={"ID":"a50cc35a-ab6b-4924-a712-2f44b4413c9d","Type":"ContainerStarted","Data":"a8cc55ee5fefe3035e5cbcfb1691c9b10ccefe46de2629cd7345e576a534e838"}
Jul 27 20:06:01.901327 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:01.901228    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:01.931622 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:01.931536    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59" podStartSLOduration=1.9315159560000001 podStartE2EDuration="1.931515956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:01.930793986 +0000 UTC m=+128.025379450" watchObservedRunningTime="2026-07-27 20:06:01.931515956 +0000 UTC m=+128.026101417"
Jul 27 20:06:03.910433 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:03.910389    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cpd7" event={"ID":"c9cfab18-fb28-417f-b0e9-7eee826ec06f","Type":"ContainerStarted","Data":"2f31f0f2777f16958e8729cff0db5569f2e66add42981cefacc83a6de1034207"}
Jul 27 20:06:04.098732 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.098692    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:06:04.101265 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.101237    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc8d3e6e-1331-40bf-a5cf-c461f7485f39-metrics-certs\") pod \"network-metrics-daemon-c9krr\" (UID: \"cc8d3e6e-1331-40bf-a5cf-c461f7485f39\") " pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:06:04.351357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.351329    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5hkrg\""
Jul 27 20:06:04.358493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.358459    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c9krr"
Jul 27 20:06:04.513276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.513238    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c9krr"]
Jul 27 20:06:04.516527 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:04.516498    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8d3e6e_1331_40bf_a5cf_c461f7485f39.slice/crio-976f1c6d47f015c65ba2d1bb657a861707ac8fb021d8be08af113fef1a9021c4 WatchSource:0}: Error finding container 976f1c6d47f015c65ba2d1bb657a861707ac8fb021d8be08af113fef1a9021c4: Status 404 returned error can't find the container with id 976f1c6d47f015c65ba2d1bb657a861707ac8fb021d8be08af113fef1a9021c4
Jul 27 20:06:04.915560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:04.915522    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9krr" event={"ID":"cc8d3e6e-1331-40bf-a5cf-c461f7485f39","Type":"ContainerStarted","Data":"976f1c6d47f015c65ba2d1bb657a861707ac8fb021d8be08af113fef1a9021c4"}
Jul 27 20:06:05.105978 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:05.105941    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:06:05.108510 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:05.108455    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff75a7d2-5329-46e6-86bd-2a9a017a1d73-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pkxt5\" (UID: \"ff75a7d2-5329-46e6-86bd-2a9a017a1d73\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:06:05.139378 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:05.139350    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s82sn\""
Jul 27 20:06:05.143560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:05.143533    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"
Jul 27 20:06:07.131854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.131826    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5"]
Jul 27 20:06:07.135416 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:07.135386    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff75a7d2_5329_46e6_86bd_2a9a017a1d73.slice/crio-40d6a46c98cd2b9be6ce2638bf2f23645c2cbcebdd59d81b0f317b19672adabf WatchSource:0}: Error finding container 40d6a46c98cd2b9be6ce2638bf2f23645c2cbcebdd59d81b0f317b19672adabf: Status 404 returned error can't find the container with id 40d6a46c98cd2b9be6ce2638bf2f23645c2cbcebdd59d81b0f317b19672adabf
Jul 27 20:06:07.928757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.928715    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9krr" event={"ID":"cc8d3e6e-1331-40bf-a5cf-c461f7485f39","Type":"ContainerStarted","Data":"065bebb1a41b540415b2649c1c00f4768689cabb10cd401ca9a213870092b848"}
Jul 27 20:06:07.928930 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.928780    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c9krr" event={"ID":"cc8d3e6e-1331-40bf-a5cf-c461f7485f39","Type":"ContainerStarted","Data":"c59ecfcf3a05dcab04f2be7b3c7d6e10905ef4d952abfbaa27e4cae3c4b6ab36"}
Jul 27 20:06:07.929782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.929763    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5" event={"ID":"ff75a7d2-5329-46e6-86bd-2a9a017a1d73","Type":"ContainerStarted","Data":"40d6a46c98cd2b9be6ce2638bf2f23645c2cbcebdd59d81b0f317b19672adabf"}
Jul 27 20:06:07.931333 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.931305    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7cpd7" event={"ID":"c9cfab18-fb28-417f-b0e9-7eee826ec06f","Type":"ContainerStarted","Data":"b545cc62d855488302da19ff0196edbf8f33d71d43ec1713bd96d5967a92d252"}
Jul 27 20:06:07.946686 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.946640    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c9krr" podStartSLOduration=130.463675164 podStartE2EDuration="2m12.94662796s" totalImagesPullingTime="2.482952796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:07.946029583 +0000 UTC m=+134.040615048" watchObservedRunningTime="2026-07-27 20:06:07.94662796 +0000 UTC m=+134.041213425"
Jul 27 20:06:07.968304 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:07.968256    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7cpd7" podStartSLOduration=17.869791602 podStartE2EDuration="29.968244205s" totalImagesPullingTime="12.098452603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:05:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:07.967593874 +0000 UTC m=+134.062179337" watchObservedRunningTime="2026-07-27 20:06:07.968244205 +0000 UTC m=+134.062829668"
Jul 27 20:06:10.943397 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:10.943357    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5" event={"ID":"ff75a7d2-5329-46e6-86bd-2a9a017a1d73","Type":"ContainerStarted","Data":"b243a98f1f699f3025ae556738d3a9c6c35d871a1dadbc25dc0df8e336a5bcd6"}
Jul 27 20:06:22.909675 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:22.909643    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d459c79d7-c5g59"
Jul 27 20:06:22.934158 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:22.934102    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pkxt5" podStartSLOduration=46.812922691 podStartE2EDuration="49.934085941s" totalImagesPullingTime="3.12116325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:10.965941908 +0000 UTC m=+137.060527373" watchObservedRunningTime="2026-07-27 20:06:22.934085941 +0000 UTC m=+149.028671407"
Jul 27 20:06:28.330474 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.330432    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"]
Jul 27 20:06:28.337565 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.337525    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.340026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.339981    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:06:28.340026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.339992    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 20:06:28.340697 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.340672    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:06:28.340807 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.340701    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5f6mt\""
Jul 27 20:06:28.346230 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.345724    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"]
Jul 27 20:06:28.377770 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.377727    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"]
Jul 27 20:06:28.382102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.382082    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ftwjr"]
Jul 27 20:06:28.382222 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.382186    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.385230 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.385209    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 20:06:28.385666 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.385644    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dn7gw\""
Jul 27 20:06:28.385763 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.385696    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 20:06:28.386208 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.386192    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:06:28.386279 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.386240    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.387769 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387749    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.387873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387788    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.387873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387838    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.387873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387863    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-textfile\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387886    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-wtmp\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387914    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.388017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387953    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.388017 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.387978    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qfz\" (UniqueName: \"kubernetes.io/projected/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-kube-api-access-l9qfz\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.388223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388070    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-root\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388102    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch92b\" (UniqueName: \"kubernetes.io/projected/ae35d615-3039-4984-9e30-9f000f79fd8e-kube-api-access-ch92b\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388131    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.388223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388168    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.388223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388216    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388239    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlzd\" (UniqueName: \"kubernetes.io/projected/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-api-access-cqlzd\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388261    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-sys\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388282    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388324    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388352    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388457 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388374    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-metrics-client-ca\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.388779 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388567    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:06:28.388779 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388579    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gvc48\""
Jul 27 20:06:28.388876 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388807    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:06:28.388937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.388923    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:06:28.402841 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.402821    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"]
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489593    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489661    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489695    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qfz\" (UniqueName: \"kubernetes.io/projected/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-kube-api-access-l9qfz\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489724    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-root\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489756    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch92b\" (UniqueName: \"kubernetes.io/projected/ae35d615-3039-4984-9e30-9f000f79fd8e-kube-api-access-ch92b\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489784    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489829    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489860    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489888    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlzd\" (UniqueName: \"kubernetes.io/projected/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-api-access-cqlzd\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489916    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-sys\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489940    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.489990    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490025    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490055    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-metrics-client-ca\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.490738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490081    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490122    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490207    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490234    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-textfile\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490282    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-wtmp\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490458    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-wtmp\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:06:28.490744    2531 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:06:28.490828    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls podName:ae35d615-3039-4984-9e30-9f000f79fd8e nodeName:}" failed. No retries permitted until 2026-07-27 20:06:28.990808372 +0000 UTC m=+155.085393829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls") pod "node-exporter-ftwjr" (UID: "ae35d615-3039-4984-9e30-9f000f79fd8e") : secret "node-exporter-tls" not found
Jul 27 20:06:28.491647 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.490979    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-root\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.492026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.491668    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-textfile\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.492026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.491728    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.492026 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:06:28.491771    2531 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 20:06:28.492026 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:06:28.491823    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls podName:a2f02f18-9da8-4415-9d4b-bda5b777f7d6 nodeName:}" failed. No retries permitted until 2026-07-27 20:06:28.991807038 +0000 UTC m=+155.086392481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-m6ssv" (UID: "a2f02f18-9da8-4415-9d4b-bda5b777f7d6") : secret "openshift-state-metrics-tls" not found
Jul 27 20:06:28.492026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.491864    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.492822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.492802    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae35d615-3039-4984-9e30-9f000f79fd8e-sys\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.493348 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.493007    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.493348 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.493224    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.493958 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.493933    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.494073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.493943    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.494375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.494353    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae35d615-3039-4984-9e30-9f000f79fd8e-metrics-client-ca\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.495246 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.495222    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.495484 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.495466    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.495546 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.495474    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.501889 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.501865    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qfz\" (UniqueName: \"kubernetes.io/projected/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-kube-api-access-l9qfz\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.502041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.502023    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlzd\" (UniqueName: \"kubernetes.io/projected/84ea07f1-3acf-46c3-8448-8ca6ae2525f9-kube-api-access-cqlzd\") pod \"kube-state-metrics-56d9f944b4-mnmcb\" (UID: \"84ea07f1-3acf-46c3-8448-8ca6ae2525f9\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.503084 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.503066    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch92b\" (UniqueName: \"kubernetes.io/projected/ae35d615-3039-4984-9e30-9f000f79fd8e-kube-api-access-ch92b\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.692771 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.692693    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"
Jul 27 20:06:28.848155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.848123    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb"]
Jul 27 20:06:28.851765 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:28.851739    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ea07f1_3acf_46c3_8448_8ca6ae2525f9.slice/crio-b929a62aba02394ce902fbb97e5b1c98537c5164d0f3d50175ebef759d78b011 WatchSource:0}: Error finding container b929a62aba02394ce902fbb97e5b1c98537c5164d0f3d50175ebef759d78b011: Status 404 returned error can't find the container with id b929a62aba02394ce902fbb97e5b1c98537c5164d0f3d50175ebef759d78b011
Jul 27 20:06:28.994811 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.994728    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.994971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.994817    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.997208 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.997166    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae35d615-3039-4984-9e30-9f000f79fd8e-node-exporter-tls\") pod \"node-exporter-ftwjr\" (UID: \"ae35d615-3039-4984-9e30-9f000f79fd8e\") " pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:28.997322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.997237    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2f02f18-9da8-4415-9d4b-bda5b777f7d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m6ssv\" (UID: \"a2f02f18-9da8-4415-9d4b-bda5b777f7d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:28.997871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:28.997853    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftwjr"
Jul 27 20:06:29.000517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.000494    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb" event={"ID":"84ea07f1-3acf-46c3-8448-8ca6ae2525f9","Type":"ContainerStarted","Data":"b929a62aba02394ce902fbb97e5b1c98537c5164d0f3d50175ebef759d78b011"}
Jul 27 20:06:29.009575 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:29.009556    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae35d615_3039_4984_9e30_9f000f79fd8e.slice/crio-57d3553e833d639ef1e11399a8d2876c69a16c027f06509eee37e9c517aca994 WatchSource:0}: Error finding container 57d3553e833d639ef1e11399a8d2876c69a16c027f06509eee37e9c517aca994: Status 404 returned error can't find the container with id 57d3553e833d639ef1e11399a8d2876c69a16c027f06509eee37e9c517aca994
Jul 27 20:06:29.248926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.248846    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"
Jul 27 20:06:29.387465 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.387439    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv"]
Jul 27 20:06:29.390134 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:29.390106    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f02f18_9da8_4415_9d4b_bda5b777f7d6.slice/crio-88bdb99bceecade963bcc331c38838f02badda9ecc5a814c2edf8b8b49dd00cb WatchSource:0}: Error finding container 88bdb99bceecade963bcc331c38838f02badda9ecc5a814c2edf8b8b49dd00cb: Status 404 returned error can't find the container with id 88bdb99bceecade963bcc331c38838f02badda9ecc5a814c2edf8b8b49dd00cb
Jul 27 20:06:29.503841 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.503783    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:06:29.510419 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.510395    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.518700 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.518676    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 20:06:29.518804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.518792    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 20:06:29.518895 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.518880    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 20:06:29.519030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.518879    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vkdtm\""
Jul 27 20:06:29.519157 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.519067    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 20:06:29.519274 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.519261    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 20:06:29.519513 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.518926    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 20:06:29.519513 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.519493    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 20:06:29.520042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.519704    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 20:06:29.520042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.519704    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 20:06:29.541708 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.541672    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:06:29.600574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600541    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600590    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600620    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600644    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600674    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600699    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600774    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600813    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zzd\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600850    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600894    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600920    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600955    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.600983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.600979    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702257 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702223    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702277    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702310    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702333    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702363    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702389    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702427    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702451    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zzd\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702488    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702535    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702561    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702595    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.702693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.702619    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.703593 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.703303    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.703706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.703603    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.708053 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.708001    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.709044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.708620    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.709044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.708642    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.709044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.708934    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.709044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.708998    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.709878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.709834    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.710533 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.710309    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.710533 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.710483    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.710680 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.710527    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.711634 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.711593    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.717598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.717567    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zzd\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd\") pod \"alertmanager-main-0\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:29.829778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:29.829740    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:06:30.008195 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:30.008047    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv" event={"ID":"a2f02f18-9da8-4415-9d4b-bda5b777f7d6","Type":"ContainerStarted","Data":"6e195e02aadadf7eeb07a9e6651cb46e5346c8c7eddfaf50008957e04b5a2256"}
Jul 27 20:06:30.008195 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:30.008118    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv" event={"ID":"a2f02f18-9da8-4415-9d4b-bda5b777f7d6","Type":"ContainerStarted","Data":"a2be0aadaf93c48736faf3af21adcd6deaf49b507e33c056f51bb13fde0e5975"}
Jul 27 20:06:30.008195 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:30.008149    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv" event={"ID":"a2f02f18-9da8-4415-9d4b-bda5b777f7d6","Type":"ContainerStarted","Data":"88bdb99bceecade963bcc331c38838f02badda9ecc5a814c2edf8b8b49dd00cb"}
Jul 27 20:06:30.009577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:30.009550    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftwjr" event={"ID":"ae35d615-3039-4984-9e30-9f000f79fd8e","Type":"ContainerStarted","Data":"57d3553e833d639ef1e11399a8d2876c69a16c027f06509eee37e9c517aca994"}
Jul 27 20:06:30.109030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:30.109003    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:06:30.379258 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:30.379169    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b361a02_e702_4af5_86d1_93718b43bf54.slice/crio-6d4e64706a6518d5b1bf2391c15fcd0537c56682eb33fe31cebacec894d1326f WatchSource:0}: Error finding container 6d4e64706a6518d5b1bf2391c15fcd0537c56682eb33fe31cebacec894d1326f: Status 404 returned error can't find the container with id 6d4e64706a6518d5b1bf2391c15fcd0537c56682eb33fe31cebacec894d1326f
Jul 27 20:06:31.015066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.014966    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb" event={"ID":"84ea07f1-3acf-46c3-8448-8ca6ae2525f9","Type":"ContainerStarted","Data":"c3e2dc2796fca73537715ae059739334a3b99a45ac73cf607844e05193618ca2"}
Jul 27 20:06:31.015534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.015073    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb" event={"ID":"84ea07f1-3acf-46c3-8448-8ca6ae2525f9","Type":"ContainerStarted","Data":"fc5eb02103fcaa5691f6f1769ef3fe1efc902c6448a0a7e4b538f4e2dc0f3baa"}
Jul 27 20:06:31.015534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.015108    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb" event={"ID":"84ea07f1-3acf-46c3-8448-8ca6ae2525f9","Type":"ContainerStarted","Data":"e157c99f772a3faf3cfb60015d1c5c70f341e1e83dade537017886354373707a"}
Jul 27 20:06:31.017072 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.017038    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv" event={"ID":"a2f02f18-9da8-4415-9d4b-bda5b777f7d6","Type":"ContainerStarted","Data":"98dcee0480c4c335ee06aa31777cf2fad695f2496e7ddcb90d2d2c52c62789f5"}
Jul 27 20:06:31.018591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.018567    2531 generic.go:356] "Generic (PLEG): container finished" podID="ae35d615-3039-4984-9e30-9f000f79fd8e" containerID="93902bc21ab949a41d52d263d5d2a1b11f6f6e7136b99e48db24e10cf72b443c" exitCode=0
Jul 27 20:06:31.018710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.018647    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftwjr" event={"ID":"ae35d615-3039-4984-9e30-9f000f79fd8e","Type":"ContainerDied","Data":"93902bc21ab949a41d52d263d5d2a1b11f6f6e7136b99e48db24e10cf72b443c"}
Jul 27 20:06:31.019991 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.019952    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"6d4e64706a6518d5b1bf2391c15fcd0537c56682eb33fe31cebacec894d1326f"}
Jul 27 20:06:31.040565 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.040510    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mnmcb" podStartSLOduration=1.2558710450000001 podStartE2EDuration="3.040491519s" totalImagesPullingTime="1.784620474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:31.039318841 +0000 UTC m=+157.133904307" watchObservedRunningTime="2026-07-27 20:06:31.040491519 +0000 UTC m=+157.135076983"
Jul 27 20:06:31.090318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.090265    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m6ssv" podStartSLOduration=1.919810647 podStartE2EDuration="3.09024395s" totalImagesPullingTime="1.170433303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:31.089270233 +0000 UTC m=+157.183855697" watchObservedRunningTime="2026-07-27 20:06:31.09024395 +0000 UTC m=+157.184829414"
Jul 27 20:06:31.458475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.458442    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-79cf5d867-h8w7r"]
Jul 27 20:06:31.462592 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.462566    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.466280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466220    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 20:06:31.466389 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466265    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 20:06:31.467170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466554    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 20:06:31.467170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466654    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 20:06:31.467170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466889    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x28mc\""
Jul 27 20:06:31.467170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.466954    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-57s8735fm3kjg\""
Jul 27 20:06:31.467170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.467140    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 20:06:31.489622 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.489534    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79cf5d867-h8w7r"]
Jul 27 20:06:31.519407 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519374    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519582 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519419    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519582 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519449    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519582 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519481    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c192b7bd-ae2b-4523-ae0e-d149444bd64a-metrics-client-ca\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519582 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519528    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvs8h\" (UniqueName: \"kubernetes.io/projected/c192b7bd-ae2b-4523-ae0e-d149444bd64a-kube-api-access-jvs8h\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519582 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519576    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519623    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.519799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.519649    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-grpc-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.620871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.620831    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvs8h\" (UniqueName: \"kubernetes.io/projected/c192b7bd-ae2b-4523-ae0e-d149444bd64a-kube-api-access-jvs8h\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.620871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.620879    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.620913    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.620939    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-grpc-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.621020    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.621046    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.621082    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.621352 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.621113    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c192b7bd-ae2b-4523-ae0e-d149444bd64a-metrics-client-ca\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.622446 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.622391    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c192b7bd-ae2b-4523-ae0e-d149444bd64a-metrics-client-ca\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.624345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.624298    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.625004 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.624690    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.625004 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.624967    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.625162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.625025    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-grpc-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.625485 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.625459    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.625719 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.625684    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c192b7bd-ae2b-4523-ae0e-d149444bd64a-secret-thanos-querier-tls\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.633850 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.633827    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvs8h\" (UniqueName: \"kubernetes.io/projected/c192b7bd-ae2b-4523-ae0e-d149444bd64a-kube-api-access-jvs8h\") pod \"thanos-querier-79cf5d867-h8w7r\" (UID: \"c192b7bd-ae2b-4523-ae0e-d149444bd64a\") " pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.774652 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.774629    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:31.903888 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:31.903856    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-79cf5d867-h8w7r"]
Jul 27 20:06:31.906775 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:31.906748    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc192b7bd_ae2b_4523_ae0e_d149444bd64a.slice/crio-bde52982aabd6ef77e496a3c4d32a7d36368cb3dd34fa4ff8dc278cbbcc978c5 WatchSource:0}: Error finding container bde52982aabd6ef77e496a3c4d32a7d36368cb3dd34fa4ff8dc278cbbcc978c5: Status 404 returned error can't find the container with id bde52982aabd6ef77e496a3c4d32a7d36368cb3dd34fa4ff8dc278cbbcc978c5
Jul 27 20:06:32.025368 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.025334    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182" exitCode=0
Jul 27 20:06:32.025766 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.025410    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"}
Jul 27 20:06:32.026650 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.026628    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"bde52982aabd6ef77e496a3c4d32a7d36368cb3dd34fa4ff8dc278cbbcc978c5"}
Jul 27 20:06:32.028626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.028599    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftwjr" event={"ID":"ae35d615-3039-4984-9e30-9f000f79fd8e","Type":"ContainerStarted","Data":"f676a8ebcc879b7ab5e478254b039e19d36557eac9b32c0aeb67be016fd566d3"}
Jul 27 20:06:32.028722 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.028647    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftwjr" event={"ID":"ae35d615-3039-4984-9e30-9f000f79fd8e","Type":"ContainerStarted","Data":"a23c2fdcdc9da1671a9b5610f1f2abd03128d1c9afc9af88ae629ba141ba644e"}
Jul 27 20:06:32.080819 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:32.080775    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ftwjr" podStartSLOduration=3.037089409 podStartE2EDuration="4.080760084s" totalImagesPullingTime="975.318652ms" totalInitContainerRuntime="68.352023ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:32.079354764 +0000 UTC m=+158.173940228" watchObservedRunningTime="2026-07-27 20:06:32.080760084 +0000 UTC m=+158.175345547"
Jul 27 20:06:33.100145 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.099691    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:06:33.150162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.150113    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:06:33.150336 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.150234    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.155241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.155031    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k7t6n\""
Jul 27 20:06:33.157450 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.157332    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:06:33.159667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.158953    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:06:33.159667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.159487    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:06:33.160114 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.159677    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:06:33.160659 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.160609    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:06:33.160803 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.160781    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:06:33.161595 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.161424    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:06:33.187807 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.187613    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:06:33.236199 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236144    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236214    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236259    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236284    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236307    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236359    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.236635 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.236394    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvq2\" (UniqueName: \"kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337649 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337604    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337666    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvq2\" (UniqueName: \"kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337778    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337806    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337843    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337864    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.337995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.337885    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.338863 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.338569    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.338863 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.338569    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.339211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.339156    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.339400 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.339378    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.341146 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.341104    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.341923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.341901    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.348531 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.348487    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvq2\" (UniqueName: \"kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2\") pod \"console-5dfd479576-jrx45\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") " pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.464277 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:33.464192    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:33.793424 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:06:33.793378    2531 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xmzlx" podUID="26082a5d-26b7-4e73-8131-f5446e7d9441"
Jul 27 20:06:34.039380 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.039347    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:34.733289 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.733265    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:06:34.756303 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.756283    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.759160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.759139    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:06:34.759394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.759375    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4bk0tmnln1n9b\""
Jul 27 20:06:34.759903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.759886    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:06:34.759985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.759950    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:06:34.766756 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.766661    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:06:34.766892 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.766850    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:06:34.767551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.767020    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:06:34.767551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.767229    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:06:34.767551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.767382    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:06:34.767948 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.767926    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.768277    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.769415    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2rqh6\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.770279    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.770497    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.770679    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.770835    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:06:34.771475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.771242    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:06:34.773437 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.773351    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:06:34.776108 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.775100    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:06:34.850361 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850334    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850368    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850391    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850452    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850503    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lks\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850550    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850599    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850620    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850688    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850719    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850751    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850776    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.850829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850815    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850846    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850868    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850893    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850914    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850943    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.850966    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.851089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.851021    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951418    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951455    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951479    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951509    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951537    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951580    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951606    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951653    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951699    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951728    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.951765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951764    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951793    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951818    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lks\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951844    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951882    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951912    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951966    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.951991    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.952021    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.952252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.952045    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.953037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.952818    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.953037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.953000    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.953037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.953009    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.954663 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.953644    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.954663 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.954097    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.954663 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.954380    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.955147 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.955105    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.955504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.955480    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.956049 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.956022    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.957162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.957143    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.957522 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.957482    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.957715 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.957690    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.957793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.957764    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.957984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.957960    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.958257 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.958227    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.958399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.958378    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.958611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.958591    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.959633 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.959615    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.960425 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.960405    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:34.970540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:34.970518    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lks\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks\") pod \"prometheus-k8s-0\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:35.046570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.046405    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"}
Jul 27 20:06:35.046570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.046522    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"}
Jul 27 20:06:35.046570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.046568    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"}
Jul 27 20:06:35.048070 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.048037    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfd479576-jrx45" event={"ID":"0d2fc18c-602f-4acd-8081-49e45150a0c3","Type":"ContainerStarted","Data":"b0a54c2b424de5bd6a5478fad4f8ff4cd491434bceb4418ce6453c00b30b3afb"}
Jul 27 20:06:35.051332 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.051303    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"d729fad0d223c4b4779d381a9b77fa203f09f9b114a6005731ead9003309d516"}
Jul 27 20:06:35.051401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.051378    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"41afa3376a0bb0e8e852123777acff148b93d9e38a4a5ce5532e71a28c194cdb"}
Jul 27 20:06:35.051444 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.051423    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"93be54948a89aeafa0f86225fd42e91470aac35817458018df37be66cebf3434"}
Jul 27 20:06:35.071126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.071105    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:35.260822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:35.260796    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:06:35.263467 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:35.263325    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805363bb_287d_4109_b8d0_6ed248fadf93.slice/crio-2df6bea81731bf743d85cff986753f9ad27f350f36f06b119ef98399b7e5f5ec WatchSource:0}: Error finding container 2df6bea81731bf743d85cff986753f9ad27f350f36f06b119ef98399b7e5f5ec: Status 404 returned error can't find the container with id 2df6bea81731bf743d85cff986753f9ad27f350f36f06b119ef98399b7e5f5ec
Jul 27 20:06:36.061366 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:36.061164    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"}
Jul 27 20:06:36.061366 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:36.061306    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"}
Jul 27 20:06:36.063321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:36.063098    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c" exitCode=0
Jul 27 20:06:36.063321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:36.063160    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"}
Jul 27 20:06:36.063321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:36.063238    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"2df6bea81731bf743d85cff986753f9ad27f350f36f06b119ef98399b7e5f5ec"}
Jul 27 20:06:37.073041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.072935    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerStarted","Data":"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"}
Jul 27 20:06:37.076844 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.076813    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"32c6ffd185663a13d947a66fce7ae7251befcdecb941810db89a9cd5a60a42a2"}
Jul 27 20:06:37.076985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.076885    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"e679c0e0dd7d9027fc2ed164ba28afb4bfbc27e02d50856aaaa5da5de212fc97"}
Jul 27 20:06:37.076985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.076932    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:37.076985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.076974    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" event={"ID":"c192b7bd-ae2b-4523-ae0e-d149444bd64a","Type":"ContainerStarted","Data":"c6ab1202c4abb06d1f1a70e8d5f0633d6333ed1e9fbbc80948dc47a5f4a843ce"}
Jul 27 20:06:37.110387 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.108991    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.059074722 podStartE2EDuration="8.108969932s" totalImagesPullingTime="5.035154123s" totalInitContainerRuntime="14.741087ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:37.104128809 +0000 UTC m=+163.198714287" watchObservedRunningTime="2026-07-27 20:06:37.108969932 +0000 UTC m=+163.203555397"
Jul 27 20:06:37.162550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:37.162493    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r" podStartSLOduration=2.074963819 podStartE2EDuration="6.162473461s" totalImagesPullingTime="4.087509642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:37.161237771 +0000 UTC m=+163.255823239" watchObservedRunningTime="2026-07-27 20:06:37.162473461 +0000 UTC m=+163.257058925"
Jul 27 20:06:38.695836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:38.695791    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:38.698585 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:38.698557    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26082a5d-26b7-4e73-8131-f5446e7d9441-metrics-tls\") pod \"dns-default-xmzlx\" (UID: \"26082a5d-26b7-4e73-8131-f5446e7d9441\") " pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:38.843737 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:38.843703    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k4bq\""
Jul 27 20:06:38.850239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:38.850213    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:39.088773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:39.088732    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfd479576-jrx45" event={"ID":"0d2fc18c-602f-4acd-8081-49e45150a0c3","Type":"ContainerStarted","Data":"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"}
Jul 27 20:06:39.113783 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:39.113729    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dfd479576-jrx45" podStartSLOduration=2.6185918790000002 podStartE2EDuration="6.11371051s" totalImagesPullingTime="3.495118631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:39.11008153 +0000 UTC m=+165.204666993" watchObservedRunningTime="2026-07-27 20:06:39.11371051 +0000 UTC m=+165.208295973"
Jul 27 20:06:39.694658 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:39.694630    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmzlx"]
Jul 27 20:06:39.696481 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:06:39.696451    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26082a5d_26b7_4e73_8131_f5446e7d9441.slice/crio-ed49160511eb60d53c3a045e9e0dc4554557ed917c502f8bcb06c132909329f0 WatchSource:0}: Error finding container ed49160511eb60d53c3a045e9e0dc4554557ed917c502f8bcb06c132909329f0: Status 404 returned error can't find the container with id ed49160511eb60d53c3a045e9e0dc4554557ed917c502f8bcb06c132909329f0
Jul 27 20:06:40.096240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096127    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"}
Jul 27 20:06:40.096240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096227    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"}
Jul 27 20:06:40.096459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096271    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"}
Jul 27 20:06:40.096459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096310    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"}
Jul 27 20:06:40.096459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096346    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"}
Jul 27 20:06:40.096459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.096383    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerStarted","Data":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"}
Jul 27 20:06:40.097347 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.097315    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzlx" event={"ID":"26082a5d-26b7-4e73-8131-f5446e7d9441","Type":"ContainerStarted","Data":"ed49160511eb60d53c3a045e9e0dc4554557ed917c502f8bcb06c132909329f0"}
Jul 27 20:06:40.136070 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:40.136014    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.559678276 podStartE2EDuration="6.135997478s" totalImagesPullingTime="3.546776349s" totalInitContainerRuntime="29.542853ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:40.13310487 +0000 UTC m=+166.227690335" watchObservedRunningTime="2026-07-27 20:06:40.135997478 +0000 UTC m=+166.230582972"
Jul 27 20:06:42.106805 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:42.106772    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzlx" event={"ID":"26082a5d-26b7-4e73-8131-f5446e7d9441","Type":"ContainerStarted","Data":"4737b2ec79b9a0ef1ec2dcbd268d25159584af241106401ea867b278bc6b2237"}
Jul 27 20:06:42.107261 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:42.106822    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmzlx" event={"ID":"26082a5d-26b7-4e73-8131-f5446e7d9441","Type":"ContainerStarted","Data":"590cf505b5c8633d6a74f991164f779be52ac25f61627430d4c1bfd2ab9f91fb"}
Jul 27 20:06:42.107261 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:42.106866    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:42.126094 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:42.126037    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xmzlx" podStartSLOduration=130.310646617 podStartE2EDuration="2m12.126023192s" totalImagesPullingTime="1.815376575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:06:42.125428069 +0000 UTC m=+168.220013556" watchObservedRunningTime="2026-07-27 20:06:42.126023192 +0000 UTC m=+168.220608655"
Jul 27 20:06:43.087298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:43.087272    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-79cf5d867-h8w7r"
Jul 27 20:06:43.464496 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:43.464465    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:43.464864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:43.464546    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:43.469586 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:43.469567    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:44.122991 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:44.122955    2531 generic.go:356] "Generic (PLEG): container finished" podID="5209a37f-3e8b-4166-a6a7-df67e9a2db38" containerID="de83cab6745c3b3ea7bdd062ea44dadaa1aae5523d0152796af2ca49c99cdf15" exitCode=0
Jul 27 20:06:44.123155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:44.123031    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv" event={"ID":"5209a37f-3e8b-4166-a6a7-df67e9a2db38","Type":"ContainerDied","Data":"de83cab6745c3b3ea7bdd062ea44dadaa1aae5523d0152796af2ca49c99cdf15"}
Jul 27 20:06:44.123487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:44.123467    2531 scope.go:117] "RemoveContainer" containerID="de83cab6745c3b3ea7bdd062ea44dadaa1aae5523d0152796af2ca49c99cdf15"
Jul 27 20:06:44.127295 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:44.127272    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:06:45.071907 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:45.071873    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:06:45.129159 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:45.129115    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv" event={"ID":"5209a37f-3e8b-4166-a6a7-df67e9a2db38","Type":"ContainerStarted","Data":"4068988abab6ef52a3afda5bc5b5516a6881b0030635012d03b0112c08b6bcb4"}
Jul 27 20:06:52.113338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:52.113258    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmzlx"
Jul 27 20:06:59.188117 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:59.188083    2531 generic.go:356] "Generic (PLEG): container finished" podID="03a10cc3-eb6e-4725-bc99-b490924fa229" containerID="c1cc8da62077870a74ebc2bd98ff0de79d4ea2cf21dd9132ba210bf14276d6e4" exitCode=0
Jul 27 20:06:59.188604 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:59.188163    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f24xp" event={"ID":"03a10cc3-eb6e-4725-bc99-b490924fa229","Type":"ContainerDied","Data":"c1cc8da62077870a74ebc2bd98ff0de79d4ea2cf21dd9132ba210bf14276d6e4"}
Jul 27 20:06:59.188696 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:06:59.188627    2531 scope.go:117] "RemoveContainer" containerID="c1cc8da62077870a74ebc2bd98ff0de79d4ea2cf21dd9132ba210bf14276d6e4"
Jul 27 20:07:00.193712 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:00.193676    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f24xp" event={"ID":"03a10cc3-eb6e-4725-bc99-b490924fa229","Type":"ContainerStarted","Data":"f03285d1ee13f5d493b7f8822620417eb4455b30baf8fceb2e9042ede6aade0f"}
Jul 27 20:07:35.072189 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:35.072150    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:35.090942 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:35.090916    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:35.358376 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:35.358303    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:48.772796 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.772756    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:48.773268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773045    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="alertmanager" containerID="cri-o://b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708" gracePeriod=120
Jul 27 20:07:48.773268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773194    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-web" containerID="cri-o://37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed" gracePeriod=120
Jul 27 20:07:48.773268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773220    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="config-reloader" containerID="cri-o://bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d" gracePeriod=120
Jul 27 20:07:48.773456 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773135    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-metric" containerID="cri-o://d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e" gracePeriod=120
Jul 27 20:07:48.773456 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773230    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy" containerID="cri-o://2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330" gracePeriod=120
Jul 27 20:07:48.773456 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:48.773223    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="prom-label-proxy" containerID="cri-o://1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170" gracePeriod=120
Jul 27 20:07:49.407782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407751    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170" exitCode=0
Jul 27 20:07:49.407782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407775    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330" exitCode=0
Jul 27 20:07:49.407782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407783    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d" exitCode=0
Jul 27 20:07:49.407782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407788    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708" exitCode=0
Jul 27 20:07:49.408042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407820    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"}
Jul 27 20:07:49.408042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407871    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"}
Jul 27 20:07:49.408042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407896    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"}
Jul 27 20:07:49.408042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:49.407918    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"}
Jul 27 20:07:50.013099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.013073    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.046191 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046130    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046191 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046186    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046220    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046238    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7zzd\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046272    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046358    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046394    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046768 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046584    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:50.046768 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046703    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.046961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046912    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.047073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.046991    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.047073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047031    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.047073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047057    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.047318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047071    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:50.047318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047086    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out\") pod \"6b361a02-e702-4af5-86d1-93718b43bf54\" (UID: \"6b361a02-e702-4af5-86d1-93718b43bf54\") "
Jul 27 20:07:50.047487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047398    2531 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-metrics-client-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.047487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.047419    2531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.050861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.050516    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.050861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.050702    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.050861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.050734    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd" (OuterVolumeSpecName: "kube-api-access-r7zzd") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "kube-api-access-r7zzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:50.051105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.050903    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.051105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.050997    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:50.051314 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.051290    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out" (OuterVolumeSpecName: "config-out") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:50.052415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.052382    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume" (OuterVolumeSpecName: "config-volume") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.053881 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.053834    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:50.054056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.054035    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.056198 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.056066    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.067470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.067440    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config" (OuterVolumeSpecName: "web-config") pod "6b361a02-e702-4af5-86d1-93718b43bf54" (UID: "6b361a02-e702-4af5-86d1-93718b43bf54"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147870    2531 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-tls-assets\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147910    2531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-alertmanager-main-db\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147921    2531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147929    2531 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b361a02-e702-4af5-86d1-93718b43bf54-config-out\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147937    2531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-main-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147946    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7zzd\" (UniqueName: \"kubernetes.io/projected/6b361a02-e702-4af5-86d1-93718b43bf54-kube-api-access-r7zzd\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147955    2531 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-config-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147962    2531 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-cluster-tls-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.147966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147970    2531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.148369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147979    2531 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-web-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.148369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.147987    2531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b361a02-e702-4af5-86d1-93718b43bf54-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:50.414862 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414771    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e" exitCode=0
Jul 27 20:07:50.414862 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414799    2531 generic.go:356] "Generic (PLEG): container finished" podID="6b361a02-e702-4af5-86d1-93718b43bf54" containerID="37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed" exitCode=0
Jul 27 20:07:50.414862 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414858    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.415100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414858    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"}
Jul 27 20:07:50.415100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414946    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"}
Jul 27 20:07:50.415100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.414985    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b361a02-e702-4af5-86d1-93718b43bf54","Type":"ContainerDied","Data":"6d4e64706a6518d5b1bf2391c15fcd0537c56682eb33fe31cebacec894d1326f"}
Jul 27 20:07:50.415100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.415012    2531 scope.go:117] "RemoveContainer" containerID="1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"
Jul 27 20:07:50.422918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.422899    2531 scope.go:117] "RemoveContainer" containerID="d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"
Jul 27 20:07:50.430120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.430081    2531 scope.go:117] "RemoveContainer" containerID="2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"
Jul 27 20:07:50.435274 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.435245    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:50.438748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.438730    2531 scope.go:117] "RemoveContainer" containerID="37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"
Jul 27 20:07:50.439311 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.439292    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:50.445788 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.445772    2531 scope.go:117] "RemoveContainer" containerID="bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"
Jul 27 20:07:50.452781 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.452764    2531 scope.go:117] "RemoveContainer" containerID="b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"
Jul 27 20:07:50.460329 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.460309    2531 scope.go:117] "RemoveContainer" containerID="c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"
Jul 27 20:07:50.466112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466085    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:50.466443 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466425    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="init-config-reloader"
Jul 27 20:07:50.466443 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466442    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="init-config-reloader"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466455    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-web"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466464    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-web"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466486    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="alertmanager"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466495    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="alertmanager"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466510    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466518    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466528    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-metric"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466537    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-metric"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466550    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="prom-label-proxy"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466558    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="prom-label-proxy"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466569    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="config-reloader"
Jul 27 20:07:50.466617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466579    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="config-reloader"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466686    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="config-reloader"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466699    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-web"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466707    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="prom-label-proxy"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466716    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy-metric"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466727    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="alertmanager"
Jul 27 20:07:50.467154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.466735    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" containerName="kube-rbac-proxy"
Jul 27 20:07:50.468113 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.468094    2531 scope.go:117] "RemoveContainer" containerID="1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"
Jul 27 20:07:50.468437 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.468415    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170\": container with ID starting with 1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170 not found: ID does not exist" containerID="1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"
Jul 27 20:07:50.468508 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.468445    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"} err="failed to get container status \"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170\": rpc error: code = NotFound desc = could not find container \"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170\": container with ID starting with 1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170 not found: ID does not exist"
Jul 27 20:07:50.468508 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.468475    2531 scope.go:117] "RemoveContainer" containerID="d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"
Jul 27 20:07:50.468747 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.468729    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e\": container with ID starting with d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e not found: ID does not exist" containerID="d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"
Jul 27 20:07:50.468783 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.468752    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"} err="failed to get container status \"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e\": rpc error: code = NotFound desc = could not find container \"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e\": container with ID starting with d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e not found: ID does not exist"
Jul 27 20:07:50.468783 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.468769    2531 scope.go:117] "RemoveContainer" containerID="2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"
Jul 27 20:07:50.469011 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.468994    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330\": container with ID starting with 2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330 not found: ID does not exist" containerID="2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"
Jul 27 20:07:50.469051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469016    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"} err="failed to get container status \"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330\": rpc error: code = NotFound desc = could not find container \"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330\": container with ID starting with 2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330 not found: ID does not exist"
Jul 27 20:07:50.469051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469030    2531 scope.go:117] "RemoveContainer" containerID="37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"
Jul 27 20:07:50.469277 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.469259    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed\": container with ID starting with 37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed not found: ID does not exist" containerID="37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"
Jul 27 20:07:50.469319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469281    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"} err="failed to get container status \"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed\": rpc error: code = NotFound desc = could not find container \"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed\": container with ID starting with 37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed not found: ID does not exist"
Jul 27 20:07:50.469319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469295    2531 scope.go:117] "RemoveContainer" containerID="bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"
Jul 27 20:07:50.469532 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.469516    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d\": container with ID starting with bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d not found: ID does not exist" containerID="bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"
Jul 27 20:07:50.469577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469535    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"} err="failed to get container status \"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d\": rpc error: code = NotFound desc = could not find container \"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d\": container with ID starting with bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d not found: ID does not exist"
Jul 27 20:07:50.469577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469547    2531 scope.go:117] "RemoveContainer" containerID="b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"
Jul 27 20:07:50.469786 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.469769    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708\": container with ID starting with b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708 not found: ID does not exist" containerID="b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"
Jul 27 20:07:50.469831 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469790    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"} err="failed to get container status \"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708\": rpc error: code = NotFound desc = could not find container \"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708\": container with ID starting with b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708 not found: ID does not exist"
Jul 27 20:07:50.469831 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.469806    2531 scope.go:117] "RemoveContainer" containerID="c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"
Jul 27 20:07:50.470000 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:50.469985    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182\": container with ID starting with c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182 not found: ID does not exist" containerID="c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"
Jul 27 20:07:50.470038 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470002    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"} err="failed to get container status \"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182\": rpc error: code = NotFound desc = could not find container \"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182\": container with ID starting with c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182 not found: ID does not exist"
Jul 27 20:07:50.470038 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470014    2531 scope.go:117] "RemoveContainer" containerID="1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"
Jul 27 20:07:50.470231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470213    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170"} err="failed to get container status \"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170\": rpc error: code = NotFound desc = could not find container \"1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170\": container with ID starting with 1ec79b69868ac5106511c3682b64635c46399af55981aa302ca55be46ea6e170 not found: ID does not exist"
Jul 27 20:07:50.470292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470233    2531 scope.go:117] "RemoveContainer" containerID="d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"
Jul 27 20:07:50.470430 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470415    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e"} err="failed to get container status \"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e\": rpc error: code = NotFound desc = could not find container \"d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e\": container with ID starting with d4704cb92e487687d3c9b8b237fa7bec9fbb471143f6cdfc2253528912573f8e not found: ID does not exist"
Jul 27 20:07:50.470495 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470432    2531 scope.go:117] "RemoveContainer" containerID="2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"
Jul 27 20:07:50.470625 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470609    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330"} err="failed to get container status \"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330\": rpc error: code = NotFound desc = could not find container \"2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330\": container with ID starting with 2ce9c2afe4d5b3efeeceffb879761e2a0f3b0007c9dbdb279593b5216baef330 not found: ID does not exist"
Jul 27 20:07:50.470665 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470626    2531 scope.go:117] "RemoveContainer" containerID="37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"
Jul 27 20:07:50.470850 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470833    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed"} err="failed to get container status \"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed\": rpc error: code = NotFound desc = could not find container \"37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed\": container with ID starting with 37eaed9d66b07e95fad22f58aa3a2439a10d9454e91fbb9a21d9f8a82c4ef0ed not found: ID does not exist"
Jul 27 20:07:50.470913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.470852    2531 scope.go:117] "RemoveContainer" containerID="bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"
Jul 27 20:07:50.471091 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.471055    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d"} err="failed to get container status \"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d\": rpc error: code = NotFound desc = could not find container \"bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d\": container with ID starting with bbca2235f1023e1a0b029651b482daf13569f4ff2d336f261fab88789c9e774d not found: ID does not exist"
Jul 27 20:07:50.471136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.471092    2531 scope.go:117] "RemoveContainer" containerID="b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"
Jul 27 20:07:50.471319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.471300    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708"} err="failed to get container status \"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708\": rpc error: code = NotFound desc = could not find container \"b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708\": container with ID starting with b5d21ff7db6d399515675dab98c5ed42b320ba1875c5116a780d455ea1103708 not found: ID does not exist"
Jul 27 20:07:50.471319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.471317    2531 scope.go:117] "RemoveContainer" containerID="c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"
Jul 27 20:07:50.471520 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.471502    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182"} err="failed to get container status \"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182\": rpc error: code = NotFound desc = could not find container \"c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182\": container with ID starting with c4a6b921eaabf30fd9750677139f664dee165a4268e768fd726c92e2d6941182 not found: ID does not exist"
Jul 27 20:07:50.472353 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.472339    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.474800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.474779    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 20:07:50.474800 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.474795    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 20:07:50.475046 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475007    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vkdtm\""
Jul 27 20:07:50.475046 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475033    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 20:07:50.475259 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475101    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 20:07:50.475346 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475304    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 20:07:50.475346 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475322    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 20:07:50.475633 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475593    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 20:07:50.475926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.475909    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 20:07:50.480343 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.480325    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 20:07:50.484396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.484373    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:50.550585 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550547    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550585 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550589    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550625    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-web-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550657    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550675    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmrb\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-kube-api-access-vxmrb\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550697    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550714    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550729    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.550802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550759    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.551005 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550801    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-out\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.551005 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550834    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.551005 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550854    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.551005 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.550879    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651643    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-web-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651680    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651702    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmrb\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-kube-api-access-vxmrb\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651720    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651737    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651754    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651794    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.651918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.651817    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-out\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.652223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652051    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.652223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652116    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.652223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652146    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.652391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652246    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.652391 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652274    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.653276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.652880    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.654762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.654732    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-out\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.654913    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.654968    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.654981    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-web-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655358 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655195    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655358 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655264    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655467 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655363    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655646 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655621    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655760    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.655905 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.655883    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c95292d-8c1a-4919-923c-9f84a18aa82f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.656633 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.656612    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c95292d-8c1a-4919-923c-9f84a18aa82f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.659384 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.659367    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmrb\" (UniqueName: \"kubernetes.io/projected/4c95292d-8c1a-4919-923c-9f84a18aa82f-kube-api-access-vxmrb\") pod \"alertmanager-main-0\" (UID: \"4c95292d-8c1a-4919-923c-9f84a18aa82f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.783621 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.783591    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:07:50.923145 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:07:50.923112    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c95292d_8c1a_4919_923c_9f84a18aa82f.slice/crio-875f33358cad96689d9e277ba34805e5f3129b4c96ca67cca3e4a63f40fd1893 WatchSource:0}: Error finding container 875f33358cad96689d9e277ba34805e5f3129b4c96ca67cca3e4a63f40fd1893: Status 404 returned error can't find the container with id 875f33358cad96689d9e277ba34805e5f3129b4c96ca67cca3e4a63f40fd1893
Jul 27 20:07:50.924585 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:50.924558    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:07:51.421410 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:51.421371    2531 generic.go:356] "Generic (PLEG): container finished" podID="4c95292d-8c1a-4919-923c-9f84a18aa82f" containerID="1adf38b00d4277714f8a6b3491ca87cc06f8e7f50361547891dc8e42fbf68268" exitCode=0
Jul 27 20:07:51.421799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:51.421460    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerDied","Data":"1adf38b00d4277714f8a6b3491ca87cc06f8e7f50361547891dc8e42fbf68268"}
Jul 27 20:07:51.421799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:51.421525    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"875f33358cad96689d9e277ba34805e5f3129b4c96ca67cca3e4a63f40fd1893"}
Jul 27 20:07:52.338109 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.338072    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b361a02-e702-4af5-86d1-93718b43bf54" path="/var/lib/kubelet/pods/6b361a02-e702-4af5-86d1-93718b43bf54/volumes"
Jul 27 20:07:52.429201 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429145    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"c083597000384391dd6efc35ba2bfdc5c315ed22f51c6f7c475518eb9b05ead5"}
Jul 27 20:07:52.429676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429244    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"b53250e04d29b161454060f50217d673df4ce24a0df79d7459be17e3ed5fbadf"}
Jul 27 20:07:52.429676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429291    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"af0442add7ba6f2af4ba87ca009523b02859b327ccdd33480e99fb37434e8a0d"}
Jul 27 20:07:52.429676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429335    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"6a3ada4dc6030acd01ebf8b201ad0303c8ead3ce10d46bcf56c6fcf527e14ebe"}
Jul 27 20:07:52.429676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429383    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"c1b2955472bf3438f2c4f191ae44561d18c09aaaff6e40bfdaaa4823bf3de18b"}
Jul 27 20:07:52.429676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.429419    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c95292d-8c1a-4919-923c-9f84a18aa82f","Type":"ContainerStarted","Data":"51f42ea60ca43d79125aa4e54e8e84754fde0a24e168c9a174873ec3c4b91a33"}
Jul 27 20:07:52.459938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:52.459886    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.441286776 podStartE2EDuration="2.459873094s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.586318ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:52.457345372 +0000 UTC m=+238.551930847" watchObservedRunningTime="2026-07-27 20:07:52.459873094 +0000 UTC m=+238.554458557"
Jul 27 20:07:53.093080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093042    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.093381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093352    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="prometheus" containerID="cri-o://13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1" gracePeriod=600
Jul 27 20:07:53.093473 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093378    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8" gracePeriod=600
Jul 27 20:07:53.093473 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093383    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="thanos-sidecar" containerID="cri-o://a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce" gracePeriod=600
Jul 27 20:07:53.093473 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093416    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-web" containerID="cri-o://f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568" gracePeriod=600
Jul 27 20:07:53.093473 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093355    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy" containerID="cri-o://538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac" gracePeriod=600
Jul 27 20:07:53.093670 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.093425    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="config-reloader" containerID="cri-o://82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0" gracePeriod=600
Jul 27 20:07:53.358496 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.358470    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.376060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376031    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376066    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376086    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376105    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lks\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376125    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376243    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376284    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376312    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376376    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376408    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376443    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376467    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376497    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376501    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376529    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376543    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.376567 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376560    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376599    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376647    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376687    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376743    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376764    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls\") pod \"805363bb-287d-4109-b8d0-6ed248fadf93\" (UID: \"805363bb-287d-4109-b8d0-6ed248fadf93\") "
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376799    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.376925    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.377086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377077    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.378140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377110    2531 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.378140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377129    2531 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-metrics-client-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.378140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377147    2531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.378140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377162    2531 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.378140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.377424    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:53.378551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.378519    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.379296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.379261    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.379587 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.379560    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.379734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.379677    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:53.380128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380095    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out" (OuterVolumeSpecName: "config-out") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:07:53.380315 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380254    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config" (OuterVolumeSpecName: "config") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.380427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380338    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks" (OuterVolumeSpecName: "kube-api-access-97lks") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "kube-api-access-97lks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:53.380427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380404    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.380590 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380565    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.380654 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380599    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:53.380723 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.380712    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.381654 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.381638    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.381916 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.381899    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.393074 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.393042    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config" (OuterVolumeSpecName: "web-config") pod "805363bb-287d-4109-b8d0-6ed248fadf93" (UID: "805363bb-287d-4109-b8d0-6ed248fadf93"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:53.438591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438555    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8" exitCode=0
Jul 27 20:07:53.438591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438588    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac" exitCode=0
Jul 27 20:07:53.438591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438597    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568" exitCode=0
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438607    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce" exitCode=0
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438616    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0" exitCode=0
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438621    2531 generic.go:356] "Generic (PLEG): container finished" podID="805363bb-287d-4109-b8d0-6ed248fadf93" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1" exitCode=0
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438660    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438656    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438832    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438868    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438897    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438904    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438943    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.438982    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"}
Jul 27 20:07:53.439140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.439039    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"805363bb-287d-4109-b8d0-6ed248fadf93","Type":"ContainerDied","Data":"2df6bea81731bf743d85cff986753f9ad27f350f36f06b119ef98399b7e5f5ec"}
Jul 27 20:07:53.447186 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.446993    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.454619 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.454599    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.462015 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.461988    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.462304 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.462290    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.467939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.467904    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.469812 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.469796    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.476998 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.476980    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477745    2531 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477769    2531 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-kube-rbac-proxy\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477786    2531 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-grpc-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477800    2531 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477814    2531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-db\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477827    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lks\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-kube-api-access-97lks\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477858    2531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477874    2531 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477902    2531 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/805363bb-287d-4109-b8d0-6ed248fadf93-config-out\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477916    2531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.477947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477929    2531 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.478472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477967    2531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.478472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477978    2531 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/805363bb-287d-4109-b8d0-6ed248fadf93-tls-assets\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.478472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477988    2531 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/805363bb-287d-4109-b8d0-6ed248fadf93-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.478472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.477999    2531 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-web-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.478472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.478022    2531 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/805363bb-287d-4109-b8d0-6ed248fadf93-secret-metrics-client-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:07:53.485448 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.485426    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.493407 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493292    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.493511 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493431    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.493682 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493667    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="config-reloader"
Jul 27 20:07:53.493682 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493683    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="config-reloader"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493692    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-thanos"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493702    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-thanos"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493712    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="thanos-sidecar"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493718    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="thanos-sidecar"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493725    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-web"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493730    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-web"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.493726    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493742    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="init-config-reloader"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493748    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="init-config-reloader"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493762    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="prometheus"
Jul 27 20:07:53.493773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493773    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="prometheus"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493753    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493782    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493788    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493787    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493840    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="config-reloader"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493848    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-web"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493855    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="prometheus"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493862    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy-thanos"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493869    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="kube-rbac-proxy"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.493890    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" containerName="thanos-sidecar"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.494040    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494061    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.494098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494076    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.494577 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.494309    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.494577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494326    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.494577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494338    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.494577 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.494563    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.494728 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494580    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.494728 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494597    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.494848 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.494830    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.494883 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494854    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.494883 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.494869    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.495119 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.495102    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.495156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495123    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.495156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495136    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.495402 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:07:53.495384    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.495458 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495405    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.495458 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495419    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.495691 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495672    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.495738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495691    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.495916 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495901    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.495916 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.495915    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.496118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496095    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.496185 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496121    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.496443 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496416    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.496505 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496445    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.496700 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496665    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.496762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496702    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.496938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496918    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.496985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.496939    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.497164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497146    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.497255 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497166    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.497412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497393    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.497459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497412    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.497622 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497606    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.497665 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497624    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.497830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497814    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.497891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.497831    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.498070 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498054    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.498111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498071    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.498278 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498261    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.498334 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498278    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.498478 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498463    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.498525 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498477    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.498671 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498654    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.498671 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498671    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.498849 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498834    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.498900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.498849    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.499060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499042    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.499060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499055    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.499152 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499060    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.499642 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499619    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.499720 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499644    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.499888 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499869    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.499949 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.499888    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.500140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500116    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.500217 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500142    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.500491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500469    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.500556 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500493    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.500731 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500712    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.500778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500733    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.500997 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500974    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.501035 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.500999    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.501340 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501318    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:07:53.501427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501327    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:07:53.501479 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501414    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.501479 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501437    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501641    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2rqh6\""
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501681    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501696    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501710    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501767    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501717    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:07:53.501821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.501793    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.502160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502056    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.502160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502078    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.502160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502099    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:07:53.502160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502105    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:07:53.502160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502066    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:07:53.502427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502350    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.502427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502371    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.502427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502387    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:07:53.502427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502399    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:07:53.502427 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502372    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4bk0tmnln1n9b\""
Jul 27 20:07:53.502657 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502640    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:07:53.502709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502636    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.502709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502668    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:07:53.502709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502679    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.502962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502935    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.502962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.502962    2531 scope.go:117] "RemoveContainer" containerID="ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"
Jul 27 20:07:53.503235 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503208    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8"} err="failed to get container status \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": rpc error: code = NotFound desc = could not find container \"ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8\": container with ID starting with ca29ae4d2af5e10af4711216f0c056a664c6bb41624de24f354cc2c3a98289a8 not found: ID does not exist"
Jul 27 20:07:53.503235 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503235    2531 scope.go:117] "RemoveContainer" containerID="538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"
Jul 27 20:07:53.503530 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503508    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac"} err="failed to get container status \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": rpc error: code = NotFound desc = could not find container \"538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac\": container with ID starting with 538af9251c4fe9563b6aff53d521ba5e2cf52e1f6ec44704891f9c8c150bc2ac not found: ID does not exist"
Jul 27 20:07:53.503584 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503535    2531 scope.go:117] "RemoveContainer" containerID="f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"
Jul 27 20:07:53.503801 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503757    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568"} err="failed to get container status \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": rpc error: code = NotFound desc = could not find container \"f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568\": container with ID starting with f31f99171be29a17b2ce58194731da575558c73353410a3c8248e170d8f59568 not found: ID does not exist"
Jul 27 20:07:53.503875 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.503803    2531 scope.go:117] "RemoveContainer" containerID="a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"
Jul 27 20:07:53.504100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504075    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce"} err="failed to get container status \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": rpc error: code = NotFound desc = could not find container \"a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce\": container with ID starting with a2c96c29cfeeb0b5dda785ca448cdec9c8edd71b7ae231b9e1f8f1aed02444ce not found: ID does not exist"
Jul 27 20:07:53.504100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504101    2531 scope.go:117] "RemoveContainer" containerID="82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"
Jul 27 20:07:53.504382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504355    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0"} err="failed to get container status \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": rpc error: code = NotFound desc = could not find container \"82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0\": container with ID starting with 82cd1cf018f344f560d6ba2ba6204d5fe00c7fae26138505a78f2d5009f28eb0 not found: ID does not exist"
Jul 27 20:07:53.504471 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504382    2531 scope.go:117] "RemoveContainer" containerID="13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"
Jul 27 20:07:53.504706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504680    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1"} err="failed to get container status \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": rpc error: code = NotFound desc = could not find container \"13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1\": container with ID starting with 13de782333e96541e6d183960e31c2e2557ae3e8bda0b4033c70d92fdaa2d5f1 not found: ID does not exist"
Jul 27 20:07:53.504808 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504707    2531 scope.go:117] "RemoveContainer" containerID="037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"
Jul 27 20:07:53.504957 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.504931    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c"} err="failed to get container status \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": rpc error: code = NotFound desc = could not find container \"037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c\": container with ID starting with 037fb77f36fb5623a9b341e6bda8d0f52a231c72cd0447b938773ec14b5c272c not found: ID does not exist"
Jul 27 20:07:53.505444 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.505428    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:07:53.516016 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.511702    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:07:53.517279 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.517256    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.579108 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579069    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579108 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579112    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579132    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579148    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579166    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579208    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579258    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579351    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579402    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579431    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579461    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579500    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579528    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579557    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579585    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579605    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579623    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579639    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579653    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49rd\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-kube-api-access-v49rd\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.579832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.579676    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680625    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680678    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680706    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680822    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680872    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680918    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.680973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680945    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.680975    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681266 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681236    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681320 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681277    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681320 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681306    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681332    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681357    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49rd\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-kube-api-access-v49rd\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681397    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681454    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681479    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681502    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681524    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681552    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681826 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681583    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.681826 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.681721    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.682656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.682326    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.682656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.682572    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.682882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.682744    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.683598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.683201    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.683598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.683288    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.683598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.683419    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.684578 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.684542    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.684712 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.684681    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.684869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.684841    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.685211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.685168    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd36787c-1664-4eea-a5bd-a2a7715c7cde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.685555 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.685491    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.685681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.685631    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.685681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.685658    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.686351 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.686329    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.686531 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.686514    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-config\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.686604 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.686513    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.686749 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.686734    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.686839 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.686823    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd36787c-1664-4eea-a5bd-a2a7715c7cde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.690643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.690624    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49rd\" (UniqueName: \"kubernetes.io/projected/cd36787c-1664-4eea-a5bd-a2a7715c7cde-kube-api-access-v49rd\") pod \"prometheus-k8s-0\" (UID: \"cd36787c-1664-4eea-a5bd-a2a7715c7cde\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.812274 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.812226    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:07:53.959211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:53.959107    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:07:53.962493 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:07:53.962469    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd36787c_1664_4eea_a5bd_a2a7715c7cde.slice/crio-2c145d86539a56dde7cbeb2f20926795411a8a8ba0ce5df0d5e723be03bdcb0a WatchSource:0}: Error finding container 2c145d86539a56dde7cbeb2f20926795411a8a8ba0ce5df0d5e723be03bdcb0a: Status 404 returned error can't find the container with id 2c145d86539a56dde7cbeb2f20926795411a8a8ba0ce5df0d5e723be03bdcb0a
Jul 27 20:07:54.338542 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:54.338507    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805363bb-287d-4109-b8d0-6ed248fadf93" path="/var/lib/kubelet/pods/805363bb-287d-4109-b8d0-6ed248fadf93/volumes"
Jul 27 20:07:54.443860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:54.443821    2531 generic.go:356] "Generic (PLEG): container finished" podID="cd36787c-1664-4eea-a5bd-a2a7715c7cde" containerID="2588744a6b3edee092e9a545960a84fbd748f769ce649fcc84431ef71c9ecda0" exitCode=0
Jul 27 20:07:54.444316 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:54.443910    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerDied","Data":"2588744a6b3edee092e9a545960a84fbd748f769ce649fcc84431ef71c9ecda0"}
Jul 27 20:07:54.444316 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:54.443975    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"2c145d86539a56dde7cbeb2f20926795411a8a8ba0ce5df0d5e723be03bdcb0a"}
Jul 27 20:07:55.451222 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451165    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"3082c404b4341130fb6633f5ab3a833798ccf86003cb8c65874b336684169921"}
Jul 27 20:07:55.451577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451277    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"5836cb8e8b9e394c4e8f2b5c4dfd0ef56f4bf2eac1a76f23526cac7bc4ec5e28"}
Jul 27 20:07:55.451577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451309    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"80c329b100f38c27c4515711ac87459e80c3aa6e7f19079757a2f172bc0b28fe"}
Jul 27 20:07:55.451577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451334    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"eb8f94c751a1e55beb1268703f6d3a07193431fb06d134a041209f35dd8fe043"}
Jul 27 20:07:55.451577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451357    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"7bf430494e3e76a3ba0bde72c65584fe941782be6719412b21d6b6b4694f9e37"}
Jul 27 20:07:55.451577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.451380    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd36787c-1664-4eea-a5bd-a2a7715c7cde","Type":"ContainerStarted","Data":"d080df440980bea865357f6054077b03358c9e57bea4d5c1ab2619fe5c0d2102"}
Jul 27 20:07:55.485275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:55.485218    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.4667974360000002 podStartE2EDuration="2.485200731s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.403295ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:55.484918422 +0000 UTC m=+241.579503898" watchObservedRunningTime="2026-07-27 20:07:55.485200731 +0000 UTC m=+241.579786192"
Jul 27 20:07:58.812660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:07:58.812620    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:08:10.018474 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:10.018435    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:08:35.038156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.038082    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dfd479576-jrx45" podUID="0d2fc18c-602f-4acd-8081-49e45150a0c3" containerName="console" containerID="cri-o://ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526" gracePeriod=15
Jul 27 20:08:35.274452 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.274425    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:08:35.346060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.345987    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346018    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346060 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346043    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346080    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346115    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346139    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.346549 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346519    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:35.346612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346548    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config" (OuterVolumeSpecName: "console-config") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:35.346612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346563    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:35.346612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.346588    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:08:35.348529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.348495    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:08:35.348529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.348521    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:08:35.446903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.446871    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnvq2\" (UniqueName: \"kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2\") pod \"0d2fc18c-602f-4acd-8081-49e45150a0c3\" (UID: \"0d2fc18c-602f-4acd-8081-49e45150a0c3\") "
Jul 27 20:08:35.447063 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447035    2531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.447063 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447045    2531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-oauth-config\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.447063 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447056    2531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-oauth-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.447251 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447065    2531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.447251 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447073    2531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d2fc18c-602f-4acd-8081-49e45150a0c3-trusted-ca-bundle\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.447251 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.447082    2531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d2fc18c-602f-4acd-8081-49e45150a0c3-console-serving-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.449013 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.448986    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2" (OuterVolumeSpecName: "kube-api-access-cnvq2") pod "0d2fc18c-602f-4acd-8081-49e45150a0c3" (UID: "0d2fc18c-602f-4acd-8081-49e45150a0c3"). InnerVolumeSpecName "kube-api-access-cnvq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:08:35.548284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.548247    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnvq2\" (UniqueName: \"kubernetes.io/projected/0d2fc18c-602f-4acd-8081-49e45150a0c3-kube-api-access-cnvq2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:08:35.621164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.621074    2531 generic.go:356] "Generic (PLEG): container finished" podID="0d2fc18c-602f-4acd-8081-49e45150a0c3" containerID="ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526" exitCode=0
Jul 27 20:08:35.621164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.621122    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dfd479576-jrx45"
Jul 27 20:08:35.621399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.621161    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfd479576-jrx45" event={"ID":"0d2fc18c-602f-4acd-8081-49e45150a0c3","Type":"ContainerDied","Data":"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"}
Jul 27 20:08:35.621399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.621238    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dfd479576-jrx45" event={"ID":"0d2fc18c-602f-4acd-8081-49e45150a0c3","Type":"ContainerDied","Data":"b0a54c2b424de5bd6a5478fad4f8ff4cd491434bceb4418ce6453c00b30b3afb"}
Jul 27 20:08:35.621399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.621260    2531 scope.go:117] "RemoveContainer" containerID="ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"
Jul 27 20:08:35.629852 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.629835    2531 scope.go:117] "RemoveContainer" containerID="ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"
Jul 27 20:08:35.630149 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:08:35.630133    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526\": container with ID starting with ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526 not found: ID does not exist" containerID="ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"
Jul 27 20:08:35.630306 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.630159    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526"} err="failed to get container status \"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526\": rpc error: code = NotFound desc = could not find container \"ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526\": container with ID starting with ca57b3fe125110a9e03ad9e7cceb8238af4c84c21084f2e232e2209d5d382526 not found: ID does not exist"
Jul 27 20:08:35.642432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.642407    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:08:35.646154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:35.646133    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dfd479576-jrx45"]
Jul 27 20:08:36.337303 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:36.337267    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d2fc18c-602f-4acd-8081-49e45150a0c3" path="/var/lib/kubelet/pods/0d2fc18c-602f-4acd-8081-49e45150a0c3/volumes"
Jul 27 20:08:53.812651 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:53.812605    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:08:53.827829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:53.827802    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:08:54.246626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:54.246553    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:08:54.254286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:54.254260    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:08:54.257781 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:54.257757    2531 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:08:54.716080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:08:54.716052    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:11:09.153449 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.153405    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"]
Jul 27 20:11:09.153954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.153750    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2fc18c-602f-4acd-8081-49e45150a0c3" containerName="console"
Jul 27 20:11:09.153954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.153762    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2fc18c-602f-4acd-8081-49e45150a0c3" containerName="console"
Jul 27 20:11:09.153954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.153826    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2fc18c-602f-4acd-8081-49e45150a0c3" containerName="console"
Jul 27 20:11:09.156860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.156841    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.159302 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.159276    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 20:11:09.159437 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.159326    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dwbhv\""
Jul 27 20:11:09.159980 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.159964    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:11:09.166161 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.166139    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"]
Jul 27 20:11:09.293923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.293888    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6046cda3-84e4-4720-a577-223b46a0337a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.293923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.293926    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrbh\" (UniqueName: \"kubernetes.io/projected/6046cda3-84e4-4720-a577-223b46a0337a-kube-api-access-tjrbh\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.394928 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.394892    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6046cda3-84e4-4720-a577-223b46a0337a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.394928 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.394928    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrbh\" (UniqueName: \"kubernetes.io/projected/6046cda3-84e4-4720-a577-223b46a0337a-kube-api-access-tjrbh\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.395348 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.395328    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6046cda3-84e4-4720-a577-223b46a0337a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.403973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.403915    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrbh\" (UniqueName: \"kubernetes.io/projected/6046cda3-84e4-4720-a577-223b46a0337a-kube-api-access-tjrbh\") pod \"openshift-lws-operator-fd8ccff4c-96nrq\" (UID: \"6046cda3-84e4-4720-a577-223b46a0337a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.476475 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.476444    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"
Jul 27 20:11:09.602010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.601976    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq"]
Jul 27 20:11:09.604287 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:09.604260    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6046cda3_84e4_4720_a577_223b46a0337a.slice/crio-ff64c3f11d41454865e84a2b85dcccdfaa2408f9e4ebaa19d91eea4de565f6f4 WatchSource:0}: Error finding container ff64c3f11d41454865e84a2b85dcccdfaa2408f9e4ebaa19d91eea4de565f6f4: Status 404 returned error can't find the container with id ff64c3f11d41454865e84a2b85dcccdfaa2408f9e4ebaa19d91eea4de565f6f4
Jul 27 20:11:09.605773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:09.605755    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:11:10.237028 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:10.236989    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq" event={"ID":"6046cda3-84e4-4720-a577-223b46a0337a","Type":"ContainerStarted","Data":"ff64c3f11d41454865e84a2b85dcccdfaa2408f9e4ebaa19d91eea4de565f6f4"}
Jul 27 20:11:13.252487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:13.252449    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq" event={"ID":"6046cda3-84e4-4720-a577-223b46a0337a","Type":"ContainerStarted","Data":"a65bbfb27b4624de4f98010a39a8d5cfbed68f77c55d5e33d7abde1c0b23c109"}
Jul 27 20:11:13.280048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:13.279998    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-96nrq" podStartSLOduration=1.61514464 podStartE2EDuration="4.279982885s" totalImagesPullingTime="2.664838245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:13.278851278 +0000 UTC m=+439.373436754" watchObservedRunningTime="2026-07-27 20:11:13.279982885 +0000 UTC m=+439.374568349"
Jul 27 20:11:33.165361 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.165247    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"]
Jul 27 20:11:33.175604 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.175580    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.178169 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.178140    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nflwb\""
Jul 27 20:11:33.178315 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.178267    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:11:33.178381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.178273    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:11:33.178381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.178318    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:11:33.178499 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.178464    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:11:33.184487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.184465    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"]
Jul 27 20:11:33.207234 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.207208    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.207326 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.207249    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prk92\" (UniqueName: \"kubernetes.io/projected/ad0e43ef-8070-4031-a84e-b7d94145a5b7-kube-api-access-prk92\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.207367 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.207327    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.207412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.207373    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-manifests\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.207412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.207389    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-tmp\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308130    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-manifests\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308163    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-tmp\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308405 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308224    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308405 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308249    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prk92\" (UniqueName: \"kubernetes.io/projected/ad0e43ef-8070-4031-a84e-b7d94145a5b7-kube-api-access-prk92\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308405 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308291    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.308619 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.308579    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-manifests\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.310598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.310565    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.310598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.310576    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad0e43ef-8070-4031-a84e-b7d94145a5b7-tmp\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.310787 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.310756    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad0e43ef-8070-4031-a84e-b7d94145a5b7-webhook-cert\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.320049 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.320028    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prk92\" (UniqueName: \"kubernetes.io/projected/ad0e43ef-8070-4031-a84e-b7d94145a5b7-kube-api-access-prk92\") pod \"opendatahub-operator-controller-manager-67944f57b-schbl\" (UID: \"ad0e43ef-8070-4031-a84e-b7d94145a5b7\") " pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.485545 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.485475    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:33.623968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:33.623938    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"]
Jul 27 20:11:33.625877 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:33.625854    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0e43ef_8070_4031_a84e_b7d94145a5b7.slice/crio-ad670c57c84e72ddf49b4ca4f214f99d9d6319d71c1572ac342e93b192c5fabf WatchSource:0}: Error finding container ad670c57c84e72ddf49b4ca4f214f99d9d6319d71c1572ac342e93b192c5fabf: Status 404 returned error can't find the container with id ad670c57c84e72ddf49b4ca4f214f99d9d6319d71c1572ac342e93b192c5fabf
Jul 27 20:11:34.345384 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:34.345317    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl" event={"ID":"ad0e43ef-8070-4031-a84e-b7d94145a5b7","Type":"ContainerStarted","Data":"ad670c57c84e72ddf49b4ca4f214f99d9d6319d71c1572ac342e93b192c5fabf"}
Jul 27 20:11:37.360330 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:37.360285    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl" event={"ID":"ad0e43ef-8070-4031-a84e-b7d94145a5b7","Type":"ContainerStarted","Data":"6fef6017fd51c9f5285fab329deea75f7a3b71fda604878a9a4c6115b21e62b5"}
Jul 27 20:11:38.365995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:38.365957    2531 generic.go:356] "Generic (PLEG): container finished" podID="ad0e43ef-8070-4031-a84e-b7d94145a5b7" containerID="6fef6017fd51c9f5285fab329deea75f7a3b71fda604878a9a4c6115b21e62b5" exitCode=0
Jul 27 20:11:38.366411 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:38.366048    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl" event={"ID":"ad0e43ef-8070-4031-a84e-b7d94145a5b7","Type":"ContainerDied","Data":"6fef6017fd51c9f5285fab329deea75f7a3b71fda604878a9a4c6115b21e62b5"}
Jul 27 20:11:39.371997 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:39.371958    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl" event={"ID":"ad0e43ef-8070-4031-a84e-b7d94145a5b7","Type":"ContainerStarted","Data":"dfd5ea5895ab89427defbd47d63c0f4c291cac82d9552d6afc06dda0d4a00adb"}
Jul 27 20:11:39.372423 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:39.372044    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:39.402188 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:39.402134    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl" podStartSLOduration=1.897479695 podStartE2EDuration="6.402120753s" totalImagesPullingTime="3.472128313s" totalInitContainerRuntime="1.032512745s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:39.399779629 +0000 UTC m=+465.494365093" watchObservedRunningTime="2026-07-27 20:11:39.402120753 +0000 UTC m=+465.496706217"
Jul 27 20:11:50.378544 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:50.378509    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67944f57b-schbl"
Jul 27 20:11:52.822644 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.822605    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"]
Jul 27 20:11:52.826006 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.825990    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:52.830660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.830639    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:11:52.831071 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.831049    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rn2xj\""
Jul 27 20:11:52.831196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.831091    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:11:52.831196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.831108    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:11:52.831196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.831118    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:11:52.873483 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.873454    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"]
Jul 27 20:11:52.990223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.990167    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrdv\" (UniqueName: \"kubernetes.io/projected/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-kube-api-access-hqrdv\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:52.990375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.990284    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:52.990375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:52.990358    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tmp\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.091702 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.091623    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.091844 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.091707    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tmp\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.091844 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.091757    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrdv\" (UniqueName: \"kubernetes.io/projected/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-kube-api-access-hqrdv\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.091844 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:53.091760    2531 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 20:11:53.091953 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:53.091855    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs podName:cfc09cef-e6af-4195-bf0d-9ba73cc92b71 nodeName:}" failed. No retries permitted until 2026-07-27 20:11:53.591833892 +0000 UTC m=+479.686419340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs") pod "kube-auth-proxy-c9f4f89b5-mblfj" (UID: "cfc09cef-e6af-4195-bf0d-9ba73cc92b71") : secret "kube-auth-proxy-tls" not found
Jul 27 20:11:53.094003 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.093979    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tmp\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.105345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.105320    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrdv\" (UniqueName: \"kubernetes.io/projected/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-kube-api-access-hqrdv\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.221429 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.221393    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76brs"]
Jul 27 20:11:53.224871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.224850    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.227670 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.227632    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 20:11:53.227771 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.227751    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b8ff2\""
Jul 27 20:11:53.236235 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.236212    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76brs"]
Jul 27 20:11:53.395140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.395046    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.395140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.395097    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvhm\" (UniqueName: \"kubernetes.io/projected/d8262546-27c2-46a9-b3c4-f7ee53bc291a-kube-api-access-2tvhm\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.495926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.495892    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.496113 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.495938    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvhm\" (UniqueName: \"kubernetes.io/projected/d8262546-27c2-46a9-b3c4-f7ee53bc291a-kube-api-access-2tvhm\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.496113 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:53.496059    2531 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 20:11:53.496244 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:53.496119    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert podName:d8262546-27c2-46a9-b3c4-f7ee53bc291a nodeName:}" failed. No retries permitted until 2026-07-27 20:11:53.996102039 +0000 UTC m=+480.090687491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert") pod "odh-model-controller-7998c67dc4-76brs" (UID: "d8262546-27c2-46a9-b3c4-f7ee53bc291a") : secret "odh-model-controller-webhook-cert" not found
Jul 27 20:11:53.506338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.506308    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvhm\" (UniqueName: \"kubernetes.io/projected/d8262546-27c2-46a9-b3c4-f7ee53bc291a-kube-api-access-2tvhm\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:53.596935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.596901    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.599224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.599201    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc09cef-e6af-4195-bf0d-9ba73cc92b71-tls-certs\") pod \"kube-auth-proxy-c9f4f89b5-mblfj\" (UID: \"cfc09cef-e6af-4195-bf0d-9ba73cc92b71\") " pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.735854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.735785    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"
Jul 27 20:11:53.914107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:53.913993    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj"]
Jul 27 20:11:53.916890 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:53.916862    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc09cef_e6af_4195_bf0d_9ba73cc92b71.slice/crio-96a5a4a372e57043f0c729f49a161b339141a66f327ec515a59040b6ed72c679 WatchSource:0}: Error finding container 96a5a4a372e57043f0c729f49a161b339141a66f327ec515a59040b6ed72c679: Status 404 returned error can't find the container with id 96a5a4a372e57043f0c729f49a161b339141a66f327ec515a59040b6ed72c679
Jul 27 20:11:54.000621 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.000542    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:54.002832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.002813    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8262546-27c2-46a9-b3c4-f7ee53bc291a-cert\") pod \"odh-model-controller-7998c67dc4-76brs\" (UID: \"d8262546-27c2-46a9-b3c4-f7ee53bc291a\") " pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:54.131102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.131071    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-898fg"]
Jul 27 20:11:54.135638 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.135608    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.135750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.135615    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:54.139018 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.138996    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 20:11:54.149382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.149086    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-898fg"]
Jul 27 20:11:54.294698 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.294563    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-76brs"]
Jul 27 20:11:54.297595 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:54.297558    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8262546_27c2_46a9_b3c4_f7ee53bc291a.slice/crio-5c9f7b03cf4475ead5ceb776a8fe52f07e56ab522488dbd51c6fdfb685f2beb3 WatchSource:0}: Error finding container 5c9f7b03cf4475ead5ceb776a8fe52f07e56ab522488dbd51c6fdfb685f2beb3: Status 404 returned error can't find the container with id 5c9f7b03cf4475ead5ceb776a8fe52f07e56ab522488dbd51c6fdfb685f2beb3
Jul 27 20:11:54.303894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.303867    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhhc\" (UniqueName: \"kubernetes.io/projected/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-kube-api-access-7dhhc\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.304041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.303955    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.404413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.404384    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhhc\" (UniqueName: \"kubernetes.io/projected/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-kube-api-access-7dhhc\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.404601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.404443    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.408563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.408529    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 20:11:54.414641 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:54.414621    2531 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 27 20:11:54.414771 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:11:54.414681    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs podName:c89184fc-f8df-4030-8bdf-02e0f4f28cd9 nodeName:}" failed. No retries permitted until 2026-07-27 20:11:54.914662509 +0000 UTC m=+481.009247953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs") pod "model-serving-api-54d89bd5fc-898fg" (UID: "c89184fc-f8df-4030-8bdf-02e0f4f28cd9") : secret "model-serving-api-tls" not found
Jul 27 20:11:54.424092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.424068    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhhc\" (UniqueName: \"kubernetes.io/projected/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-kube-api-access-7dhhc\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:54.441196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.441158    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj" event={"ID":"cfc09cef-e6af-4195-bf0d-9ba73cc92b71","Type":"ContainerStarted","Data":"96a5a4a372e57043f0c729f49a161b339141a66f327ec515a59040b6ed72c679"}
Jul 27 20:11:54.442375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.442344    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-76brs" event={"ID":"d8262546-27c2-46a9-b3c4-f7ee53bc291a","Type":"ContainerStarted","Data":"5c9f7b03cf4475ead5ceb776a8fe52f07e56ab522488dbd51c6fdfb685f2beb3"}
Jul 27 20:11:54.807197 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.806131    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:11:54.812511 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.812480    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.816658 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.816614    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 20:11:54.816884 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.816863    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:11:54.817166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.817133    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 20:11:54.817489 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.817373    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 20:11:54.817489 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.817396    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z9jnb\""
Jul 27 20:11:54.833540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.833516    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909158    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909237    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909270    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909303    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909328    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgst\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909377    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:54.909750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:54.909415    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.010672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010627    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010684    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010718    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010744    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgst\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010780    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010804    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010855    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.010950    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.011750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.011517    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.013982 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.013878    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.014255 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.014227    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.014373 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.014236    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c89184fc-f8df-4030-8bdf-02e0f4f28cd9-tls-certs\") pod \"model-serving-api-54d89bd5fc-898fg\" (UID: \"c89184fc-f8df-4030-8bdf-02e0f4f28cd9\") " pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:55.014373 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.014318    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.014488 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.014395    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.020869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.020844    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgst\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.020970 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.020945    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lp7pl\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.065653 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.065578    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:55.125997 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.125967    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:11:55.257504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.257423    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-898fg"]
Jul 27 20:11:55.267477 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:55.267447    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89184fc_f8df_4030_8bdf_02e0f4f28cd9.slice/crio-7b602b88856b5e33d2fa7206e42660b80f6b072535aca54bee5f776ce60f0170 WatchSource:0}: Error finding container 7b602b88856b5e33d2fa7206e42660b80f6b072535aca54bee5f776ce60f0170: Status 404 returned error can't find the container with id 7b602b88856b5e33d2fa7206e42660b80f6b072535aca54bee5f776ce60f0170
Jul 27 20:11:55.332315 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.332233    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:11:55.337960 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:11:55.337927    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bb20d5_cc1c_4938_acd1_eb851846c73d.slice/crio-78d2a98ad0fbb94b091afa472ce0df6dd195dfb7eb2dd5a511796b6041cf81bc WatchSource:0}: Error finding container 78d2a98ad0fbb94b091afa472ce0df6dd195dfb7eb2dd5a511796b6041cf81bc: Status 404 returned error can't find the container with id 78d2a98ad0fbb94b091afa472ce0df6dd195dfb7eb2dd5a511796b6041cf81bc
Jul 27 20:11:55.452160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.452059    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" event={"ID":"06bb20d5-cc1c-4938-acd1-eb851846c73d","Type":"ContainerStarted","Data":"78d2a98ad0fbb94b091afa472ce0df6dd195dfb7eb2dd5a511796b6041cf81bc"}
Jul 27 20:11:55.453879 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:55.453805    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-898fg" event={"ID":"c89184fc-f8df-4030-8bdf-02e0f4f28cd9","Type":"ContainerStarted","Data":"7b602b88856b5e33d2fa7206e42660b80f6b072535aca54bee5f776ce60f0170"}
Jul 27 20:11:58.438455 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:58.438434    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:11:59.479766 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.479710    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj" event={"ID":"cfc09cef-e6af-4195-bf0d-9ba73cc92b71","Type":"ContainerStarted","Data":"52cb0cbc3fd5a1a4b93cfeaf1f12ab780ef7826b74577428ac5fdb38893c1773"}
Jul 27 20:11:59.481537 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.481507    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-898fg" event={"ID":"c89184fc-f8df-4030-8bdf-02e0f4f28cd9","Type":"ContainerStarted","Data":"d048b12ecf1cec26341a3d535c4c51421ff25da0ff1f44370657e261aac159d5"}
Jul 27 20:11:59.481654 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.481565    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:11:59.483267 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.483238    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-76brs" event={"ID":"d8262546-27c2-46a9-b3c4-f7ee53bc291a","Type":"ContainerStarted","Data":"2c25a76e0b151da6d7e692aef80529cabee6d07a7c7ff012940c395153903dd8"}
Jul 27 20:11:59.483414 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.483397    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:11:59.499237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.499189    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c9f4f89b5-mblfj" podStartSLOduration=2.98205895 podStartE2EDuration="7.499162896s" totalImagesPullingTime="4.517103946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:59.496858831 +0000 UTC m=+485.591444299" watchObservedRunningTime="2026-07-27 20:11:59.499162896 +0000 UTC m=+485.593748362"
Jul 27 20:11:59.519067 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.519006    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-898fg" podStartSLOduration=2.35292468 podStartE2EDuration="5.518986398s" totalImagesPullingTime="3.166061718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:59.517853003 +0000 UTC m=+485.612438471" watchObservedRunningTime="2026-07-27 20:11:59.518986398 +0000 UTC m=+485.613571863"
Jul 27 20:11:59.537413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.537369    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-76brs" podStartSLOduration=2.404062037 podStartE2EDuration="6.537353768s" totalImagesPullingTime="4.133291731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:59.534887901 +0000 UTC m=+485.629473384" watchObservedRunningTime="2026-07-27 20:11:59.537353768 +0000 UTC m=+485.631939280"
Jul 27 20:11:59.757463 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.757415    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:11:59.757555 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:11:59.757497    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:12:00.489269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:00.489213    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" event={"ID":"06bb20d5-cc1c-4938-acd1-eb851846c73d","Type":"ContainerStarted","Data":"9ce6dc1c7bc45f5c5233cc12ec1f92a525b32afe97613c40dd2abfde0c65ea98"}
Jul 27 20:12:00.535629 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:00.535580    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" podStartSLOduration=2.118033804 podStartE2EDuration="6.535562688s" totalImagesPullingTime="4.417528884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:00.535200825 +0000 UTC m=+486.629786308" watchObservedRunningTime="2026-07-27 20:12:00.535562688 +0000 UTC m=+486.630148153"
Jul 27 20:12:00.976155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:00.976127    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"]
Jul 27 20:12:00.980518 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:00.980492    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:00.984449 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:00.984418    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-c4prk\""
Jul 27 20:12:01.003964 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.003938    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"]
Jul 27 20:12:01.173854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.173814    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.173860    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.173888    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.173997    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.174030    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtmn\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-kube-api-access-7rtmn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.174071    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.174090    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.174111    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.174278 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.174219    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55751b8-797d-4d31-a26e-fd3c64a2be52-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.274969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.274933    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.274975    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtmn\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-kube-api-access-7rtmn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275003    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275019    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275090    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275140    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55751b8-797d-4d31-a26e-fd3c64a2be52-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275257    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275311    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275339    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275508 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275444    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275533    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275616 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275579    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275726 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275706    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.275885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.275861    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b55751b8-797d-4d31-a26e-fd3c64a2be52-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.277592 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.277568    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.277939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.277917    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.289377 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.289353    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.290089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.290071    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtmn\" (UniqueName: \"kubernetes.io/projected/b55751b8-797d-4d31-a26e-fd3c64a2be52-kube-api-access-7rtmn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66\" (UID: \"b55751b8-797d-4d31-a26e-fd3c64a2be52\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.292904 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.292885    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:01.456488 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.456435    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"]
Jul 27 20:12:01.496072 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.496039    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66" event={"ID":"b55751b8-797d-4d31-a26e-fd3c64a2be52","Type":"ContainerStarted","Data":"408ea02c4dc79561a1aa535fe8eaee8ceae6add79946aa099a3f7715f3eaa647"}
Jul 27 20:12:01.496455 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.496264    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:12:01.497942 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:01.497921    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:12:04.195049 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.195013    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"]
Jul 27 20:12:04.203398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.203372    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.208667 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.208642    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 27 20:12:04.209458 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.209439    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v5j7v\""
Jul 27 20:12:04.221319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.221292    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"]
Jul 27 20:12:04.225291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.225271    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.231321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.231298    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 20:12:04.231903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.231881    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j6gwp\""
Jul 27 20:12:04.243068 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.243041    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"]
Jul 27 20:12:04.278842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.278809    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"]
Jul 27 20:12:04.302484 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.302440    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7e60e9-c77f-4945-a32a-206f7604151f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.302484 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.302483    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvlz\" (UniqueName: \"kubernetes.io/projected/8520ebf0-6848-4f90-8ba2-a84ae16136a1-kube-api-access-jdvlz\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.302735 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.302509    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8520ebf0-6848-4f90-8ba2-a84ae16136a1-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.302735 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.302663    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8qp\" (UniqueName: \"kubernetes.io/projected/7b7e60e9-c77f-4945-a32a-206f7604151f-kube-api-access-sh8qp\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.403635 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.403600    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8qp\" (UniqueName: \"kubernetes.io/projected/7b7e60e9-c77f-4945-a32a-206f7604151f-kube-api-access-sh8qp\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.403821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.403732    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7e60e9-c77f-4945-a32a-206f7604151f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.403821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.403770    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvlz\" (UniqueName: \"kubernetes.io/projected/8520ebf0-6848-4f90-8ba2-a84ae16136a1-kube-api-access-jdvlz\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.403821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.403799    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8520ebf0-6848-4f90-8ba2-a84ae16136a1-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.406988 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.406956    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7b7e60e9-c77f-4945-a32a-206f7604151f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.407153 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.407134    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8520ebf0-6848-4f90-8ba2-a84ae16136a1-cert\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.433992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.433965    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvlz\" (UniqueName: \"kubernetes.io/projected/8520ebf0-6848-4f90-8ba2-a84ae16136a1-kube-api-access-jdvlz\") pod \"kserve-controller-manager-d6f4b65b8-pdjld\" (UID: \"8520ebf0-6848-4f90-8ba2-a84ae16136a1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.443741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.443706    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8qp\" (UniqueName: \"kubernetes.io/projected/7b7e60e9-c77f-4945-a32a-206f7604151f-kube-api-access-sh8qp\") pod \"llmisvc-controller-manager-777c6bd7d5-cmnmp\" (UID: \"7b7e60e9-c77f-4945-a32a-206f7604151f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.517510 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.517487    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:04.537902 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.537879    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:04.606748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.606359    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:12:04.606748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.606427    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:12:04.606748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.606454    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:12:04.704082 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.704051    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"]
Jul 27 20:12:04.708029 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:12:04.707950    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8520ebf0_6848_4f90_8ba2_a84ae16136a1.slice/crio-293dbd4f540e9e04682a6e8c45f9d410e645982f64def87ceff95d053293a75a WatchSource:0}: Error finding container 293dbd4f540e9e04682a6e8c45f9d410e645982f64def87ceff95d053293a75a: Status 404 returned error can't find the container with id 293dbd4f540e9e04682a6e8c45f9d410e645982f64def87ceff95d053293a75a
Jul 27 20:12:04.737869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:04.737836    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"]
Jul 27 20:12:04.742012 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:12:04.741985    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7b7e60e9_c77f_4945_a32a_206f7604151f.slice/crio-1595427273d5a121ac90df851b2a2a34c8a41c587ac783166f1814e9d85baf5b WatchSource:0}: Error finding container 1595427273d5a121ac90df851b2a2a34c8a41c587ac783166f1814e9d85baf5b: Status 404 returned error can't find the container with id 1595427273d5a121ac90df851b2a2a34c8a41c587ac783166f1814e9d85baf5b
Jul 27 20:12:05.307339 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.307302    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:12:05.307718 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.307467    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" podUID="06bb20d5-cc1c-4938-acd1-eb851846c73d" containerName="discovery" containerID="cri-o://9ce6dc1c7bc45f5c5233cc12ec1f92a525b32afe97613c40dd2abfde0c65ea98" gracePeriod=30
Jul 27 20:12:05.521277 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.520829    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld" event={"ID":"8520ebf0-6848-4f90-8ba2-a84ae16136a1","Type":"ContainerStarted","Data":"293dbd4f540e9e04682a6e8c45f9d410e645982f64def87ceff95d053293a75a"}
Jul 27 20:12:05.523683 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.523619    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66" event={"ID":"b55751b8-797d-4d31-a26e-fd3c64a2be52","Type":"ContainerStarted","Data":"84522c47d6e8a8d9c9b9272648f23e46f223a6d0b10fac5d35922aa1a75f4250"}
Jul 27 20:12:05.526408 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.526378    2531 generic.go:356] "Generic (PLEG): container finished" podID="06bb20d5-cc1c-4938-acd1-eb851846c73d" containerID="9ce6dc1c7bc45f5c5233cc12ec1f92a525b32afe97613c40dd2abfde0c65ea98" exitCode=0
Jul 27 20:12:05.526518 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.526500    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" event={"ID":"06bb20d5-cc1c-4938-acd1-eb851846c73d","Type":"ContainerDied","Data":"9ce6dc1c7bc45f5c5233cc12ec1f92a525b32afe97613c40dd2abfde0c65ea98"}
Jul 27 20:12:05.530075 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.530011    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp" event={"ID":"7b7e60e9-c77f-4945-a32a-206f7604151f","Type":"ContainerStarted","Data":"1595427273d5a121ac90df851b2a2a34c8a41c587ac783166f1814e9d85baf5b"}
Jul 27 20:12:05.548914 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.548853    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66" podStartSLOduration=2.409720806 podStartE2EDuration="5.548831633s" totalImagesPullingTime="3.139110827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:05.548217901 +0000 UTC m=+491.642803364" watchObservedRunningTime="2026-07-27 20:12:05.548831633 +0000 UTC m=+491.643417098"
Jul 27 20:12:05.634163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.634134    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:12:05.715359 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715326    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.715504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715368    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.715504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715428    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxgst\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.715504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715476    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.715504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715498    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.716115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715552    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.716115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.715626    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap\") pod \"06bb20d5-cc1c-4938-acd1-eb851846c73d\" (UID: \"06bb20d5-cc1c-4938-acd1-eb851846c73d\") "
Jul 27 20:12:05.716338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.716305    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:05.718133 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.718075    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs" (OuterVolumeSpecName: "local-certs") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:12:05.718133 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.718098    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:12:05.718133 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.718082    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:12:05.718564 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.718538    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token" (OuterVolumeSpecName: "istio-token") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:05.718733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.718715    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts" (OuterVolumeSpecName: "cacerts") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:12:05.719474 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.719448    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst" (OuterVolumeSpecName: "kube-api-access-vxgst") pod "06bb20d5-cc1c-4938-acd1-eb851846c73d" (UID: "06bb20d5-cc1c-4938-acd1-eb851846c73d"). InnerVolumeSpecName "kube-api-access-vxgst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:05.816629 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816565    2531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816629 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816615    2531 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-cacerts\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816629 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816629    2531 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/06bb20d5-cc1c-4938-acd1-eb851846c73d-local-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816642    2531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-ca-configmap\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816667    2531 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-kubeconfig\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816683    2531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/06bb20d5-cc1c-4938-acd1-eb851846c73d-istio-csr-dns-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:05.816940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:05.816696    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxgst\" (UniqueName: \"kubernetes.io/projected/06bb20d5-cc1c-4938-acd1-eb851846c73d-kube-api-access-vxgst\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:06.293884 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.293848    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:06.299510 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.299480    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:06.538813 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.538771    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl" event={"ID":"06bb20d5-cc1c-4938-acd1-eb851846c73d","Type":"ContainerDied","Data":"78d2a98ad0fbb94b091afa472ce0df6dd195dfb7eb2dd5a511796b6041cf81bc"}
Jul 27 20:12:06.538813 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.538816    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"
Jul 27 20:12:06.539346 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.538844    2531 scope.go:117] "RemoveContainer" containerID="9ce6dc1c7bc45f5c5233cc12ec1f92a525b32afe97613c40dd2abfde0c65ea98"
Jul 27 20:12:06.539346 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.539277    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:06.540545 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.540512    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66"
Jul 27 20:12:06.593206 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.593097    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:12:06.602827 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:06.602794    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lp7pl"]
Jul 27 20:12:07.547037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:07.546999    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld" event={"ID":"8520ebf0-6848-4f90-8ba2-a84ae16136a1","Type":"ContainerStarted","Data":"66796598df6ef5dfb280b24bd3d85e97569ce3aa3a1a2564fc2f04efc1c4b7d5"}
Jul 27 20:12:07.547529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:07.547103    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:07.565971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:07.565916    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld" podStartSLOduration=1.14879683 podStartE2EDuration="3.565896776s" totalImagesPullingTime="2.417099946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:07.565219368 +0000 UTC m=+493.659804833" watchObservedRunningTime="2026-07-27 20:12:07.565896776 +0000 UTC m=+493.660482244"
Jul 27 20:12:08.339937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:08.339893    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bb20d5-cc1c-4938-acd1-eb851846c73d" path="/var/lib/kubelet/pods/06bb20d5-cc1c-4938-acd1-eb851846c73d/volumes"
Jul 27 20:12:08.554566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:08.554530    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp" event={"ID":"7b7e60e9-c77f-4945-a32a-206f7604151f","Type":"ContainerStarted","Data":"4b1560749d9a611345ce6de6fd33b2d2a7f681dbad3baeccf894c3c9ad0c3263"}
Jul 27 20:12:08.582674 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:08.582514    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp" podStartSLOduration=1.3212566159999999 podStartE2EDuration="4.582495299s" totalImagesPullingTime="3.261238683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:08.58128498 +0000 UTC m=+494.675870446" watchObservedRunningTime="2026-07-27 20:12:08.582495299 +0000 UTC m=+494.677080775"
Jul 27 20:12:09.560762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:09.560731    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:10.491971 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:10.491935    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-76brs"
Jul 27 20:12:10.493921 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:10.493900    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-898fg"
Jul 27 20:12:38.559772 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:38.559736    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:12:40.567037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:40.567007    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cmnmp"
Jul 27 20:12:53.774452 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.774367    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:53.774910 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.774825    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb20d5-cc1c-4938-acd1-eb851846c73d" containerName="discovery"
Jul 27 20:12:53.774910 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.774840    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb20d5-cc1c-4938-acd1-eb851846c73d" containerName="discovery"
Jul 27 20:12:53.774979 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.774917    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bb20d5-cc1c-4938-acd1-eb851846c73d" containerName="discovery"
Jul 27 20:12:53.778361 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.778336    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:53.783274 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.783243    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:12:53.784041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.784015    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:12:53.784160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.784044    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qp74b\""
Jul 27 20:12:53.791065 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.791040    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:53.855466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.855436    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckg2\" (UniqueName: \"kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2\") pod \"kuadrant-operator-catalog-c6qhp\" (UID: \"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c\") " pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:53.955984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.955937    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckg2\" (UniqueName: \"kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2\") pod \"kuadrant-operator-catalog-c6qhp\" (UID: \"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c\") " pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:53.978424 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:53.978403    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckg2\" (UniqueName: \"kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2\") pod \"kuadrant-operator-catalog-c6qhp\" (UID: \"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c\") " pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:54.088640 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.088535    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:54.109721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.109670    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:54.222240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.222191    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:54.224522 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:12:54.224495    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ffff10_4cf5_4cc9_b1b9_016ad5f75d3c.slice/crio-566a95417b75c8a41010ae29309028c74933b7b39f054f793cd936df0275ceff WatchSource:0}: Error finding container 566a95417b75c8a41010ae29309028c74933b7b39f054f793cd936df0275ceff: Status 404 returned error can't find the container with id 566a95417b75c8a41010ae29309028c74933b7b39f054f793cd936df0275ceff
Jul 27 20:12:54.322508 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.322477    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5wl9"]
Jul 27 20:12:54.327428 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.327400    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:12:54.339687 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.339628    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5wl9"]
Jul 27 20:12:54.359285 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.359259    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjkf\" (UniqueName: \"kubernetes.io/projected/5ee86923-d8ae-4a76-9fcc-a6eb98e2423d-kube-api-access-bgjkf\") pod \"kuadrant-operator-catalog-m5wl9\" (UID: \"5ee86923-d8ae-4a76-9fcc-a6eb98e2423d\") " pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:12:54.460613 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.460562    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjkf\" (UniqueName: \"kubernetes.io/projected/5ee86923-d8ae-4a76-9fcc-a6eb98e2423d-kube-api-access-bgjkf\") pod \"kuadrant-operator-catalog-m5wl9\" (UID: \"5ee86923-d8ae-4a76-9fcc-a6eb98e2423d\") " pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:12:54.470998 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.470968    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjkf\" (UniqueName: \"kubernetes.io/projected/5ee86923-d8ae-4a76-9fcc-a6eb98e2423d-kube-api-access-bgjkf\") pod \"kuadrant-operator-catalog-m5wl9\" (UID: \"5ee86923-d8ae-4a76-9fcc-a6eb98e2423d\") " pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:12:54.639788 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.639719    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:12:54.775992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.775967    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-m5wl9"]
Jul 27 20:12:54.778593 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:12:54.778564    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee86923_d8ae_4a76_9fcc_a6eb98e2423d.slice/crio-336a06edda6d7a9f42713b9cd668105ca2c0266a867afdd95a6ea3c2d13529f1 WatchSource:0}: Error finding container 336a06edda6d7a9f42713b9cd668105ca2c0266a867afdd95a6ea3c2d13529f1: Status 404 returned error can't find the container with id 336a06edda6d7a9f42713b9cd668105ca2c0266a867afdd95a6ea3c2d13529f1
Jul 27 20:12:54.784105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:54.784080    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" event={"ID":"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c","Type":"ContainerStarted","Data":"566a95417b75c8a41010ae29309028c74933b7b39f054f793cd936df0275ceff"}
Jul 27 20:12:55.790621 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:55.790586    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9" event={"ID":"5ee86923-d8ae-4a76-9fcc-a6eb98e2423d","Type":"ContainerStarted","Data":"336a06edda6d7a9f42713b9cd668105ca2c0266a867afdd95a6ea3c2d13529f1"}
Jul 27 20:12:56.796883 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:56.796845    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9" event={"ID":"5ee86923-d8ae-4a76-9fcc-a6eb98e2423d","Type":"ContainerStarted","Data":"1155b8e0cb71fcb5d0723eb5c0a56c4f4de3644828d1a7a8987914b9c5cee393"}
Jul 27 20:12:56.798243 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:56.798218    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" event={"ID":"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c","Type":"ContainerStarted","Data":"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"}
Jul 27 20:12:56.798370 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:56.798262    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" podUID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" containerName="registry-server" containerID="cri-o://614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c" gracePeriod=2
Jul 27 20:12:56.812945 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:56.812903    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9" podStartSLOduration=1.356309813 podStartE2EDuration="2.812887444s" totalImagesPullingTime="1.456577631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:56.811780227 +0000 UTC m=+542.906365691" watchObservedRunningTime="2026-07-27 20:12:56.812887444 +0000 UTC m=+542.907472939"
Jul 27 20:12:56.827881 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:56.827838    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" podStartSLOduration=1.817242902 podStartE2EDuration="3.827823642s" totalImagesPullingTime="2.01058074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:12:56.826702947 +0000 UTC m=+542.921288425" watchObservedRunningTime="2026-07-27 20:12:56.827823642 +0000 UTC m=+542.922409104"
Jul 27 20:12:57.041966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.041943    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:57.086370 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.086302    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vckg2\" (UniqueName: \"kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2\") pod \"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c\" (UID: \"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c\") "
Jul 27 20:12:57.088298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.088276    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2" (OuterVolumeSpecName: "kube-api-access-vckg2") pod "08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" (UID: "08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c"). InnerVolumeSpecName "kube-api-access-vckg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:57.187196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.187147    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vckg2\" (UniqueName: \"kubernetes.io/projected/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c-kube-api-access-vckg2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:12:57.804320 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.804281    2531 generic.go:356] "Generic (PLEG): container finished" podID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" containerID="614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c" exitCode=0
Jul 27 20:12:57.804791 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.804328    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp"
Jul 27 20:12:57.804791 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.804370    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" event={"ID":"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c","Type":"ContainerDied","Data":"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"}
Jul 27 20:12:57.804791 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.804418    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c6qhp" event={"ID":"08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c","Type":"ContainerDied","Data":"566a95417b75c8a41010ae29309028c74933b7b39f054f793cd936df0275ceff"}
Jul 27 20:12:57.804791 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.804458    2531 scope.go:117] "RemoveContainer" containerID="614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"
Jul 27 20:12:57.814231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.814215    2531 scope.go:117] "RemoveContainer" containerID="614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"
Jul 27 20:12:57.814469 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:12:57.814454    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c\": container with ID starting with 614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c not found: ID does not exist" containerID="614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"
Jul 27 20:12:57.814528 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.814496    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c"} err="failed to get container status \"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c\": rpc error: code = NotFound desc = could not find container \"614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c\": container with ID starting with 614b96678e48cc49b31ef1c4406710ada5a559a774f999276bd3de50234ba30c not found: ID does not exist"
Jul 27 20:12:57.825027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.825005    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:57.830962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:57.830942    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c6qhp"]
Jul 27 20:12:58.338695 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:12:58.338661    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" path="/var/lib/kubelet/pods/08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c/volumes"
Jul 27 20:13:04.640757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:04.640719    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:13:04.641157 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:04.640792    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:13:04.662281 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:04.662245    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:13:04.860284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:04.860257    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-m5wl9"
Jul 27 20:13:26.906685 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.906649    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"]
Jul 27 20:13:26.907156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.907057    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" containerName="registry-server"
Jul 27 20:13:26.907156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.907068    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" containerName="registry-server"
Jul 27 20:13:26.907156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.907146    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ffff10-4cf5-4cc9-b1b9-016ad5f75d3c" containerName="registry-server"
Jul 27 20:13:26.909737 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.909721    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:26.912973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.912954    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-524lt\""
Jul 27 20:13:26.927503 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:26.927480    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"]
Jul 27 20:13:27.048643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.048607    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9nf\" (UniqueName: \"kubernetes.io/projected/f79b6153-c516-4bcb-b21f-9829d15f22d7-kube-api-access-4w9nf\") pod \"authorino-operator-75cc8d5cb7-qwdh2\" (UID: \"f79b6153-c516-4bcb-b21f-9829d15f22d7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:27.149345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.149312    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9nf\" (UniqueName: \"kubernetes.io/projected/f79b6153-c516-4bcb-b21f-9829d15f22d7-kube-api-access-4w9nf\") pod \"authorino-operator-75cc8d5cb7-qwdh2\" (UID: \"f79b6153-c516-4bcb-b21f-9829d15f22d7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:27.159697 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.159638    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9nf\" (UniqueName: \"kubernetes.io/projected/f79b6153-c516-4bcb-b21f-9829d15f22d7-kube-api-access-4w9nf\") pod \"authorino-operator-75cc8d5cb7-qwdh2\" (UID: \"f79b6153-c516-4bcb-b21f-9829d15f22d7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:27.220135 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.220108    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:27.361788 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.361762    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"]
Jul 27 20:13:27.364220 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:27.364161    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79b6153_c516_4bcb_b21f_9829d15f22d7.slice/crio-c695aa4c66fb83ed20f43303a2981f2f40447af7037a0fba3ccde4efa7a216b9 WatchSource:0}: Error finding container c695aa4c66fb83ed20f43303a2981f2f40447af7037a0fba3ccde4efa7a216b9: Status 404 returned error can't find the container with id c695aa4c66fb83ed20f43303a2981f2f40447af7037a0fba3ccde4efa7a216b9
Jul 27 20:13:27.952243 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:27.952204    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2" event={"ID":"f79b6153-c516-4bcb-b21f-9829d15f22d7","Type":"ContainerStarted","Data":"c695aa4c66fb83ed20f43303a2981f2f40447af7037a0fba3ccde4efa7a216b9"}
Jul 27 20:13:29.963549 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:29.963463    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2" event={"ID":"f79b6153-c516-4bcb-b21f-9829d15f22d7","Type":"ContainerStarted","Data":"dcac1014ff1c1e2ce1477a1300c1ddfe8cd593802459085983d3897f7fd899a5"}
Jul 27 20:13:29.963549 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:29.963537    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:29.984746 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:29.984699    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2" podStartSLOduration=1.692055022 podStartE2EDuration="3.984683047s" totalImagesPullingTime="2.292628025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:29.982490021 +0000 UTC m=+576.077075484" watchObservedRunningTime="2026-07-27 20:13:29.984683047 +0000 UTC m=+576.079268550"
Jul 27 20:13:30.078779 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.078741    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"]
Jul 27 20:13:30.086814 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.086789    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:30.089791 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.089773    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s9dhd\""
Jul 27 20:13:30.097461 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.097438    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"]
Jul 27 20:13:30.279494 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.279468    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89ng\" (UniqueName: \"kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng\") pod \"limitador-operator-controller-manager-859d7b78bd-z55cg\" (UID: \"296e2cca-6cb4-4eba-8975-d12c74e8c523\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:30.380120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.380091    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89ng\" (UniqueName: \"kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng\") pod \"limitador-operator-controller-manager-859d7b78bd-z55cg\" (UID: \"296e2cca-6cb4-4eba-8975-d12c74e8c523\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:30.390769 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.390740    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89ng\" (UniqueName: \"kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng\") pod \"limitador-operator-controller-manager-859d7b78bd-z55cg\" (UID: \"296e2cca-6cb4-4eba-8975-d12c74e8c523\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:30.397554 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.397532    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:30.532974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.532949    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"]
Jul 27 20:13:30.535008 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:30.534981    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296e2cca_6cb4_4eba_8975_d12c74e8c523.slice/crio-b2a04fe7b161dfbf05640a47b5b131263b2ac576e85a98274d73b90660c166a6 WatchSource:0}: Error finding container b2a04fe7b161dfbf05640a47b5b131263b2ac576e85a98274d73b90660c166a6: Status 404 returned error can't find the container with id b2a04fe7b161dfbf05640a47b5b131263b2ac576e85a98274d73b90660c166a6
Jul 27 20:13:30.970482 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:30.970389    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg" event={"ID":"296e2cca-6cb4-4eba-8975-d12c74e8c523","Type":"ContainerStarted","Data":"b2a04fe7b161dfbf05640a47b5b131263b2ac576e85a98274d73b90660c166a6"}
Jul 27 20:13:32.684628 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.684595    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"]
Jul 27 20:13:32.688441 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.688417    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:32.692035 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.692020    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 20:13:32.692259 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.692244    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s22jd\""
Jul 27 20:13:32.701893 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.701846    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gtx\" (UniqueName: \"kubernetes.io/projected/330952c1-0596-4b64-9903-42f9756a8d5e-kube-api-access-g6gtx\") pod \"dns-operator-controller-manager-864dd78c4c-f9j5x\" (UID: \"330952c1-0596-4b64-9903-42f9756a8d5e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:32.704874 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.704853    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"]
Jul 27 20:13:32.803006 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.802972    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gtx\" (UniqueName: \"kubernetes.io/projected/330952c1-0596-4b64-9903-42f9756a8d5e-kube-api-access-g6gtx\") pod \"dns-operator-controller-manager-864dd78c4c-f9j5x\" (UID: \"330952c1-0596-4b64-9903-42f9756a8d5e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:32.818037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.818003    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gtx\" (UniqueName: \"kubernetes.io/projected/330952c1-0596-4b64-9903-42f9756a8d5e-kube-api-access-g6gtx\") pod \"dns-operator-controller-manager-864dd78c4c-f9j5x\" (UID: \"330952c1-0596-4b64-9903-42f9756a8d5e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:32.983690 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.983601    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg" event={"ID":"296e2cca-6cb4-4eba-8975-d12c74e8c523","Type":"ContainerStarted","Data":"7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb"}
Jul 27 20:13:32.983824 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.983686    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:32.997869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:32.997847    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:33.006962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:33.006913    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg" podStartSLOduration=1.424310252 podStartE2EDuration="3.006897076s" totalImagesPullingTime="1.582586824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:33.006712396 +0000 UTC m=+579.101297860" watchObservedRunningTime="2026-07-27 20:13:33.006897076 +0000 UTC m=+579.101482542"
Jul 27 20:13:33.128555 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:33.128532    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"]
Jul 27 20:13:33.130903 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:33.130877    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330952c1_0596_4b64_9903_42f9756a8d5e.slice/crio-7274ed85b1127f539ced5f6a44932fa05b43fb059aee7960231058c39fc799b1 WatchSource:0}: Error finding container 7274ed85b1127f539ced5f6a44932fa05b43fb059aee7960231058c39fc799b1: Status 404 returned error can't find the container with id 7274ed85b1127f539ced5f6a44932fa05b43fb059aee7960231058c39fc799b1
Jul 27 20:13:33.992462 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:33.992379    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x" event={"ID":"330952c1-0596-4b64-9903-42f9756a8d5e","Type":"ContainerStarted","Data":"7274ed85b1127f539ced5f6a44932fa05b43fb059aee7960231058c39fc799b1"}
Jul 27 20:13:36.009643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:36.009609    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x" event={"ID":"330952c1-0596-4b64-9903-42f9756a8d5e","Type":"ContainerStarted","Data":"90fb4b975a46c91fe7cc0395d7a6f1468fd3f38c616989cd0d04c109b9bd4092"}
Jul 27 20:13:36.010066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:36.009807    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:36.030409 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:36.030365    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x" podStartSLOduration=2.024690848 podStartE2EDuration="4.030353983s" totalImagesPullingTime="2.005663135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:36.028022658 +0000 UTC m=+582.122608121" watchObservedRunningTime="2026-07-27 20:13:36.030353983 +0000 UTC m=+582.124939446"
Jul 27 20:13:37.293433 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.293396    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"]
Jul 27 20:13:37.297292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.297269    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.301260 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.301240    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m4xvk\""
Jul 27 20:13:37.312281 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.312260    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"]
Jul 27 20:13:37.344724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.344698    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.344861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.344839    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ftc\" (UniqueName: \"kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.445385 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.445353    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.445536 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.445463    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ftc\" (UniqueName: \"kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.445729 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.445710    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.468572 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.468543    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ftc\" (UniqueName: \"kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.607193 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.607097    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:37.736750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:37.736726    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"]
Jul 27 20:13:37.738983 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:37.738958    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa96181a_e380_4849_bd76_12f5d627feee.slice/crio-fc1d0d9896b1422d01e7c83632b682aa5aee11c2f8941972001036420a16ea63 WatchSource:0}: Error finding container fc1d0d9896b1422d01e7c83632b682aa5aee11c2f8941972001036420a16ea63: Status 404 returned error can't find the container with id fc1d0d9896b1422d01e7c83632b682aa5aee11c2f8941972001036420a16ea63
Jul 27 20:13:38.020570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:38.020534    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r" event={"ID":"fa96181a-e380-4849-bd76-12f5d627feee","Type":"ContainerStarted","Data":"fc1d0d9896b1422d01e7c83632b682aa5aee11c2f8941972001036420a16ea63"}
Jul 27 20:13:40.972378 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:40.972346    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qwdh2"
Jul 27 20:13:43.995500 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:43.995471    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:46.067918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:46.067874    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r" event={"ID":"fa96181a-e380-4849-bd76-12f5d627feee","Type":"ContainerStarted","Data":"f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3"}
Jul 27 20:13:46.068386 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:46.068002    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:46.088400 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:46.088351    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r" podStartSLOduration=1.576665437 podStartE2EDuration="9.088335999s" totalImagesPullingTime="7.511670562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:46.086292097 +0000 UTC m=+592.180877573" watchObservedRunningTime="2026-07-27 20:13:46.088335999 +0000 UTC m=+592.182921464"
Jul 27 20:13:47.017213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:47.017162    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f9j5x"
Jul 27 20:13:54.284968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:54.284943    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:13:54.286772 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:54.286730    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:13:57.076813 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:57.076781    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:58.120637 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.120579    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"]
Jul 27 20:13:58.124228 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.124209    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.141283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.141250    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"]
Jul 27 20:13:58.241556 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.241511    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2th9\" (UniqueName: \"kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.241556 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.241561    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.342342 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.342301    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2th9\" (UniqueName: \"kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.342524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.342355    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.342794 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.342773    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.360822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.360791    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2th9\" (UniqueName: \"kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.434710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.434620    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:58.576893 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.576866    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"]
Jul 27 20:13:58.578669 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:58.578640    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2656c6dd_49b8_44b0_a3ae_773473e21548.slice/crio-15e8e2c49ca747a9da8a9d3b53d957b4bad6d4ff7d5c4a4cb9ecdf8c8c8acc63 WatchSource:0}: Error finding container 15e8e2c49ca747a9da8a9d3b53d957b4bad6d4ff7d5c4a4cb9ecdf8c8c8acc63: Status 404 returned error can't find the container with id 15e8e2c49ca747a9da8a9d3b53d957b4bad6d4ff7d5c4a4cb9ecdf8c8c8acc63
Jul 27 20:13:58.835166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.835128    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"]
Jul 27 20:13:58.841847 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.841799    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"]
Jul 27 20:13:58.851823 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.851799    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"]
Jul 27 20:13:58.851998 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.851978    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r" podUID="fa96181a-e380-4849-bd76-12f5d627feee" containerName="manager" containerID="cri-o://f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3" gracePeriod=2
Jul 27 20:13:58.865511 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.865485    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"]
Jul 27 20:13:58.870544 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.870520    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:13:58.870993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.870976    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" containerName="manager"
Jul 27 20:13:58.870993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.870994    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" containerName="manager"
Jul 27 20:13:58.871122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.871005    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa96181a-e380-4849-bd76-12f5d627feee" containerName="manager"
Jul 27 20:13:58.871122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.871011    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa96181a-e380-4849-bd76-12f5d627feee" containerName="manager"
Jul 27 20:13:58.871122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.871091    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa96181a-e380-4849-bd76-12f5d627feee" containerName="manager"
Jul 27 20:13:58.871122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.871103    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" containerName="manager"
Jul 27 20:13:58.871553 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.871518    2531 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa96181a-e380-4849-bd76-12f5d627feee\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6n65r\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:58.874280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.874264    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:58.876541 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.876519    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:58.880751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.880725    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"]
Jul 27 20:13:58.880951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.880930    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" containerName="manager" containerID="cri-o://7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb" gracePeriod=2
Jul 27 20:13:58.884886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.884861    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:13:58.889929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.889878    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:58.891280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.891259    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"]
Jul 27 20:13:58.895141 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.895116    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:13:58.903554 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.903531    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:13:58.911816 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.911788    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"]
Jul 27 20:13:58.912325 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.912307    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" containerName="manager"
Jul 27 20:13:58.912325 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.912326    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" containerName="manager"
Jul 27 20:13:58.912709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.912692    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" containerName="manager"
Jul 27 20:13:58.913016 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.912983    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:58.915291 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.915263    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:58.915834 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.915821    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:13:58.934061 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.933990    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"]
Jul 27 20:13:58.947612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.947584    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9gn\" (UniqueName: \"kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:58.947766 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:58.947631    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:58.964912 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.964881    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:58.967137 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:58.967111    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.049031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.048994    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2px\" (UniqueName: \"kubernetes.io/projected/2b70cfd5-2380-4b0c-9ec0-417a4a301732-kube-api-access-ch2px\") pod \"limitador-operator-controller-manager-859d7b78bd-6sn2t\" (UID: \"2b70cfd5-2380-4b0c-9ec0-417a4a301732\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:13:59.049209 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.049132    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9gn\" (UniqueName: \"kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:59.049313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.049293    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzcx2\" (UniqueName: \"kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.049378 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.049364    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:59.049508 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.049482    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.049816 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.049791    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:59.064042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.063954    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9gn\" (UniqueName: \"kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ddrws\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:59.095068 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.095042    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.097274 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.097238    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.099323 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.099289    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.133187 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.133146    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.135410 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.135374    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.137295 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.137259    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.150656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.150629    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzcx2\" (UniqueName: \"kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.150767 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.150673    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.151293 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.151235    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2px\" (UniqueName: \"kubernetes.io/projected/2b70cfd5-2380-4b0c-9ec0-417a4a301732-kube-api-access-ch2px\") pod \"limitador-operator-controller-manager-859d7b78bd-6sn2t\" (UID: \"2b70cfd5-2380-4b0c-9ec0-417a4a301732\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:13:59.151449 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.151426    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.159089 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.159062    2531 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6" container="manager"
Jul 27 20:13:59.160398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.160375    2531 generic.go:356] "Generic (PLEG): container finished" podID="fa96181a-e380-4849-bd76-12f5d627feee" containerID="f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3" exitCode=0
Jul 27 20:13:59.160512 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.160428    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.160512 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.160472    2531 scope.go:117] "RemoveContainer" containerID="f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3"
Jul 27 20:13:59.161396 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.161366    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.161521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.161479    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzcx2\" (UniqueName: \"kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ww2ln\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.161607 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.161584    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2px\" (UniqueName: \"kubernetes.io/projected/2b70cfd5-2380-4b0c-9ec0-417a4a301732-kube-api-access-ch2px\") pod \"limitador-operator-controller-manager-859d7b78bd-6sn2t\" (UID: \"2b70cfd5-2380-4b0c-9ec0-417a4a301732\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:13:59.162128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.162110    2531 generic.go:356] "Generic (PLEG): container finished" podID="296e2cca-6cb4-4eba-8975-d12c74e8c523" containerID="7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb" exitCode=0
Jul 27 20:13:59.162221 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.162166    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.163347 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.163315    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:59.165284 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.165258    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.167142 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.167122    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:59.169404 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.169380    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.170395 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.170381    2531 scope.go:117] "RemoveContainer" containerID="f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3"
Jul 27 20:13:59.170624 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.170608    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3\": container with ID starting with f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3 not found: ID does not exist" containerID="f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3"
Jul 27 20:13:59.170672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.170630    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3"} err="failed to get container status \"f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3\": rpc error: code = NotFound desc = could not find container \"f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3\": container with ID starting with f7ae93ae94e31a4b053c745fd1fcb37e78537ef557dcc94f6f1af16f4737d9a3 not found: ID does not exist"
Jul 27 20:13:59.170672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.170645    2531 scope.go:117] "RemoveContainer" containerID="7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb"
Jul 27 20:13:59.171064 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.171045    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.181476 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.181450    2531 scope.go:117] "RemoveContainer" containerID="7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb"
Jul 27 20:13:59.181739 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.181718    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb\": container with ID starting with 7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb not found: ID does not exist" containerID="7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb"
Jul 27 20:13:59.181793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.181744    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb"} err="failed to get container status \"7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb\": rpc error: code = NotFound desc = could not find container \"7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb\": container with ID starting with 7bd4f1a6e1e4d7570d428532a7d55f1d7b2de2a54e280d96bfb777cee07c96fb not found: ID does not exist"
Jul 27 20:13:59.252413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.252380    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume\") pod \"fa96181a-e380-4849-bd76-12f5d627feee\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") "
Jul 27 20:13:59.252573 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.252427    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5ftc\" (UniqueName: \"kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc\") pod \"fa96181a-e380-4849-bd76-12f5d627feee\" (UID: \"fa96181a-e380-4849-bd76-12f5d627feee\") "
Jul 27 20:13:59.252573 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.252509    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x89ng\" (UniqueName: \"kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng\") pod \"296e2cca-6cb4-4eba-8975-d12c74e8c523\" (UID: \"296e2cca-6cb4-4eba-8975-d12c74e8c523\") "
Jul 27 20:13:59.252820 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.252792    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fa96181a-e380-4849-bd76-12f5d627feee" (UID: "fa96181a-e380-4849-bd76-12f5d627feee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:13:59.254509 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.254490    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc" (OuterVolumeSpecName: "kube-api-access-g5ftc") pod "fa96181a-e380-4849-bd76-12f5d627feee" (UID: "fa96181a-e380-4849-bd76-12f5d627feee"). InnerVolumeSpecName "kube-api-access-g5ftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:13:59.254593 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.254518    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng" (OuterVolumeSpecName: "kube-api-access-x89ng") pod "296e2cca-6cb4-4eba-8975-d12c74e8c523" (UID: "296e2cca-6cb4-4eba-8975-d12c74e8c523"). InnerVolumeSpecName "kube-api-access-x89ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:13:59.280725 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.280684    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:13:59.288701 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.288679    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:13:59.296706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.296682    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:13:59.353825 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.353776    2531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa96181a-e380-4849-bd76-12f5d627feee-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:13:59.353936 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.353828    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ftc\" (UniqueName: \"kubernetes.io/projected/fa96181a-e380-4849-bd76-12f5d627feee-kube-api-access-g5ftc\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:13:59.353936 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.353846    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89ng\" (UniqueName: \"kubernetes.io/projected/296e2cca-6cb4-4eba-8975-d12c74e8c523-kube-api-access-x89ng\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:13:59.443918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.443890    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:13:59.445834 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:59.445809    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb1c55c_561f_42a9_89ee_45765a7c4dc5.slice/crio-f90d982f0a70d2fc74149cdc3b875f17a566efd1915f2a178d06451ca5767ccc WatchSource:0}: Error finding container f90d982f0a70d2fc74149cdc3b875f17a566efd1915f2a178d06451ca5767ccc: Status 404 returned error can't find the container with id f90d982f0a70d2fc74149cdc3b875f17a566efd1915f2a178d06451ca5767ccc
Jul 27 20:13:59.498293 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.498263    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.500268 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.500240    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:59.502196 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.502145    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.505425 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.505401    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:13:59.508514 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.508486    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:13:59.510788 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:13:59.510764    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:13:59.690231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.690208    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"]
Jul 27 20:13:59.693536 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:59.693501    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b70cfd5_2380_4b0c_9ec0_417a4a301732.slice/crio-f9d5d271a811c32c9bd3a467cac7b6f031f0d10fe85ad3f8540f75f97236287c WatchSource:0}: Error finding container f9d5d271a811c32c9bd3a467cac7b6f031f0d10fe85ad3f8540f75f97236287c: Status 404 returned error can't find the container with id f9d5d271a811c32c9bd3a467cac7b6f031f0d10fe85ad3f8540f75f97236287c
Jul 27 20:13:59.694939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:13:59.694918    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:13:59.697815 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:13:59.697792    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74fbe2f_9ac0_453e_b5d4_dcc248f7336a.slice/crio-af6991597c400204e367bb4e4b7e1a0cbba51782b0e5702f50f01d6f24dcaa50 WatchSource:0}: Error finding container af6991597c400204e367bb4e4b7e1a0cbba51782b0e5702f50f01d6f24dcaa50: Status 404 returned error can't find the container with id af6991597c400204e367bb4e4b7e1a0cbba51782b0e5702f50f01d6f24dcaa50
Jul 27 20:14:00.169823 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.169774    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" event={"ID":"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a","Type":"ContainerStarted","Data":"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"}
Jul 27 20:14:00.170329 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.169843    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" event={"ID":"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a","Type":"ContainerStarted","Data":"af6991597c400204e367bb4e4b7e1a0cbba51782b0e5702f50f01d6f24dcaa50"}
Jul 27 20:14:00.170329 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.169868    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:14:00.171566 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.171531    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" containerName="manager" containerID="cri-o://5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48" gracePeriod=2
Jul 27 20:14:00.172105 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.172079    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:00.173999 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.173969    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t" event={"ID":"2b70cfd5-2380-4b0c-9ec0-417a4a301732","Type":"ContainerStarted","Data":"a6178c02927766b3b9bf7e24f1ebaa0cab0275d7ad149a072a6a87a55f25f5ef"}
Jul 27 20:14:00.174102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.174020    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t" event={"ID":"2b70cfd5-2380-4b0c-9ec0-417a4a301732","Type":"ContainerStarted","Data":"f9d5d271a811c32c9bd3a467cac7b6f031f0d10fe85ad3f8540f75f97236287c"}
Jul 27 20:14:00.174102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.174044    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:14:00.174102 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.173971    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:14:00.175923 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.175899    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:14:00.176446 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.176424    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" event={"ID":"7cb1c55c-561f-42a9-89ee-45765a7c4dc5","Type":"ContainerStarted","Data":"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"}
Jul 27 20:14:00.176538 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.176464    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" event={"ID":"7cb1c55c-561f-42a9-89ee-45765a7c4dc5","Type":"ContainerStarted","Data":"f90d982f0a70d2fc74149cdc3b875f17a566efd1915f2a178d06451ca5767ccc"}
Jul 27 20:14:00.176538 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.176488    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:14:00.202489 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.202441    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" podStartSLOduration=2.202426788 podStartE2EDuration="2.202426788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:00.200592206 +0000 UTC m=+606.295177671" watchObservedRunningTime="2026-07-27 20:14:00.202426788 +0000 UTC m=+606.297012251"
Jul 27 20:14:00.203240 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.203215    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z55cg\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z55cg"
Jul 27 20:14:00.204842 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.204820    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6n65r\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="fa96181a-e380-4849-bd76-12f5d627feee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6n65r"
Jul 27 20:14:00.231036 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.230896    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" podStartSLOduration=2.230878419 podStartE2EDuration="2.230878419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:00.229910692 +0000 UTC m=+606.324496158" watchObservedRunningTime="2026-07-27 20:14:00.230878419 +0000 UTC m=+606.325463885"
Jul 27 20:14:00.232080 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:00.232051    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:00.255816 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.255759    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t" podStartSLOduration=2.255738693 podStartE2EDuration="2.255738693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:00.253822169 +0000 UTC m=+606.348407644" watchObservedRunningTime="2026-07-27 20:14:00.255738693 +0000 UTC m=+606.350324157"
Jul 27 20:14:00.342671 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.342639    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296e2cca-6cb4-4eba-8975-d12c74e8c523" path="/var/lib/kubelet/pods/296e2cca-6cb4-4eba-8975-d12c74e8c523/volumes"
Jul 27 20:14:00.343135 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.343116    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa96181a-e380-4849-bd76-12f5d627feee" path="/var/lib/kubelet/pods/fa96181a-e380-4849-bd76-12f5d627feee/volumes"
Jul 27 20:14:00.423332 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.423277    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:00.567472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.567442    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2th9\" (UniqueName: \"kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9\") pod \"2656c6dd-49b8-44b0-a3ae-773473e21548\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") "
Jul 27 20:14:00.567633 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.567484    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume\") pod \"2656c6dd-49b8-44b0-a3ae-773473e21548\" (UID: \"2656c6dd-49b8-44b0-a3ae-773473e21548\") "
Jul 27 20:14:00.567907 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.567885    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2656c6dd-49b8-44b0-a3ae-773473e21548" (UID: "2656c6dd-49b8-44b0-a3ae-773473e21548"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:14:00.569487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.569462    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9" (OuterVolumeSpecName: "kube-api-access-c2th9") pod "2656c6dd-49b8-44b0-a3ae-773473e21548" (UID: "2656c6dd-49b8-44b0-a3ae-773473e21548"). InnerVolumeSpecName "kube-api-access-c2th9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:00.669055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.669017    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2th9\" (UniqueName: \"kubernetes.io/projected/2656c6dd-49b8-44b0-a3ae-773473e21548-kube-api-access-c2th9\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:00.669055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:00.669047    2531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2656c6dd-49b8-44b0-a3ae-773473e21548-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:01.183092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:01.183057    2531 generic.go:356] "Generic (PLEG): container finished" podID="2656c6dd-49b8-44b0-a3ae-773473e21548" containerID="5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48" exitCode=0
Jul 27 20:14:01.183551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:01.183094    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:01.183551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:01.183154    2531 scope.go:117] "RemoveContainer" containerID="5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48"
Jul 27 20:14:01.185664 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:01.185637    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:01.192695 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:01.192679    2531 scope.go:117] "RemoveContainer" containerID="5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48"
Jul 27 20:14:01.192953 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:01.192935    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48\": container with ID starting with 5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48 not found: ID does not exist" containerID="5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48"
Jul 27 20:14:01.193008 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:01.192961    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48"} err="failed to get container status \"5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48\": rpc error: code = NotFound desc = could not find container \"5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48\": container with ID starting with 5b713c76956caad0cb332d11888d1b8ad51c33d1f39c338fb6917428e99eaf48 not found: ID does not exist"
Jul 27 20:14:01.193771 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:01.193748    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p8ng6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p8ng6"
Jul 27 20:14:02.339663 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:02.339627    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2656c6dd-49b8-44b0-a3ae-773473e21548" path="/var/lib/kubelet/pods/2656c6dd-49b8-44b0-a3ae-773473e21548/volumes"
Jul 27 20:14:11.185633 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.185597    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:14:11.186025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.185679    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6sn2t"
Jul 27 20:14:11.186192 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.186159    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:14:11.278298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.278262    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:14:11.278514 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.278487    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" podUID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" containerName="manager" containerID="cri-o://ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2" gracePeriod=10
Jul 27 20:14:11.516974 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.516952    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:14:11.554037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.553999    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:14:11.554620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.554599    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" containerName="manager"
Jul 27 20:14:11.554620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.554623    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" containerName="manager"
Jul 27 20:14:11.554811 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.554766    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" containerName="manager"
Jul 27 20:14:11.558345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.558326    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.576346 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.576321    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:14:11.670998 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.670963    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume\") pod \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") "
Jul 27 20:14:11.671163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.671033    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzcx2\" (UniqueName: \"kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2\") pod \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\" (UID: \"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a\") "
Jul 27 20:14:11.671278 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.671254    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.671334 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.671311    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcnq\" (UniqueName: \"kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.671458 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.671427    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" (UID: "f74fbe2f-9ac0-453e-b5d4-dcc248f7336a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:14:11.673096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.673061    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2" (OuterVolumeSpecName: "kube-api-access-vzcx2") pod "f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" (UID: "f74fbe2f-9ac0-453e-b5d4-dcc248f7336a"). InnerVolumeSpecName "kube-api-access-vzcx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:11.772635 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.772598    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.772798 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.772652    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcnq\" (UniqueName: \"kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.772798 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.772716    2531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:11.772798 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.772731    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzcx2\" (UniqueName: \"kubernetes.io/projected/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a-kube-api-access-vzcx2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:11.772976 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.772956    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.782286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.782257    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcnq\" (UniqueName: \"kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq\") pod \"kuadrant-operator-controller-manager-86878c769f-wkbbd\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:11.868616 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:11.868584    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:12.004216 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.004187    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:14:12.006774 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:14:12.006748    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bba92c_0784_4133_a132_d1f335a63f9b.slice/crio-dfc793081f4e6be171ac6a2550a1d86bd855126174eb2b10618aa383044c35d7 WatchSource:0}: Error finding container dfc793081f4e6be171ac6a2550a1d86bd855126174eb2b10618aa383044c35d7: Status 404 returned error can't find the container with id dfc793081f4e6be171ac6a2550a1d86bd855126174eb2b10618aa383044c35d7
Jul 27 20:14:12.252318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.252281    2531 generic.go:356] "Generic (PLEG): container finished" podID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" containerID="ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2" exitCode=0
Jul 27 20:14:12.252773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.252357    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"
Jul 27 20:14:12.252773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.252381    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" event={"ID":"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a","Type":"ContainerDied","Data":"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"}
Jul 27 20:14:12.252773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.252444    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln" event={"ID":"f74fbe2f-9ac0-453e-b5d4-dcc248f7336a","Type":"ContainerDied","Data":"af6991597c400204e367bb4e4b7e1a0cbba51782b0e5702f50f01d6f24dcaa50"}
Jul 27 20:14:12.252773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.252473    2531 scope.go:117] "RemoveContainer" containerID="ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"
Jul 27 20:14:12.254500 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.254466    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" event={"ID":"09bba92c-0784-4133-a132-d1f335a63f9b","Type":"ContainerStarted","Data":"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"}
Jul 27 20:14:12.254620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.254513    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" event={"ID":"09bba92c-0784-4133-a132-d1f335a63f9b","Type":"ContainerStarted","Data":"dfc793081f4e6be171ac6a2550a1d86bd855126174eb2b10618aa383044c35d7"}
Jul 27 20:14:12.254620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.254541    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:12.261705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.261685    2531 scope.go:117] "RemoveContainer" containerID="ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"
Jul 27 20:14:12.261950 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:12.261929    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2\": container with ID starting with ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2 not found: ID does not exist" containerID="ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"
Jul 27 20:14:12.262014 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.261963    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2"} err="failed to get container status \"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2\": rpc error: code = NotFound desc = could not find container \"ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2\": container with ID starting with ec3de0d2bf04cea327079a3b05e5f02fbe00a437258f2833f54a7f7d57738cd2 not found: ID does not exist"
Jul 27 20:14:12.296100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.296003    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" podStartSLOduration=1.295987075 podStartE2EDuration="1.295987075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:12.294821927 +0000 UTC m=+618.389407390" watchObservedRunningTime="2026-07-27 20:14:12.295987075 +0000 UTC m=+618.390572539"
Jul 27 20:14:12.340586 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.340563    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:14:12.342107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:12.342085    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ww2ln"]
Jul 27 20:14:14.339099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:14.339067    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74fbe2f-9ac0-453e-b5d4-dcc248f7336a" path="/var/lib/kubelet/pods/f74fbe2f-9ac0-453e-b5d4-dcc248f7336a/volumes"
Jul 27 20:14:23.264372 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.264296    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:14:23.325753 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.324433    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:14:23.325753 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.324818    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" podUID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" containerName="manager" containerID="cri-o://d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c" gracePeriod=10
Jul 27 20:14:23.576766 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.576741    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:14:23.678885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.678851    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms9gn\" (UniqueName: \"kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn\") pod \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") "
Jul 27 20:14:23.679073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.678984    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume\") pod \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\" (UID: \"7cb1c55c-561f-42a9-89ee-45765a7c4dc5\") "
Jul 27 20:14:23.679373 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.679348    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7cb1c55c-561f-42a9-89ee-45765a7c4dc5" (UID: "7cb1c55c-561f-42a9-89ee-45765a7c4dc5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:14:23.680910 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.680890    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn" (OuterVolumeSpecName: "kube-api-access-ms9gn") pod "7cb1c55c-561f-42a9-89ee-45765a7c4dc5" (UID: "7cb1c55c-561f-42a9-89ee-45765a7c4dc5"). InnerVolumeSpecName "kube-api-access-ms9gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:23.779940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.779900    2531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:23.779940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:23.779932    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9gn\" (UniqueName: \"kubernetes.io/projected/7cb1c55c-561f-42a9-89ee-45765a7c4dc5-kube-api-access-ms9gn\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:24.333309 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.333271    2531 generic.go:356] "Generic (PLEG): container finished" podID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" containerID="d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c" exitCode=0
Jul 27 20:14:24.334891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.334874    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"
Jul 27 20:14:24.339138 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.339108    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" event={"ID":"7cb1c55c-561f-42a9-89ee-45765a7c4dc5","Type":"ContainerDied","Data":"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"}
Jul 27 20:14:24.339264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.339152    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws" event={"ID":"7cb1c55c-561f-42a9-89ee-45765a7c4dc5","Type":"ContainerDied","Data":"f90d982f0a70d2fc74149cdc3b875f17a566efd1915f2a178d06451ca5767ccc"}
Jul 27 20:14:24.339264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.339214    2531 scope.go:117] "RemoveContainer" containerID="d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"
Jul 27 20:14:24.348743 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.348725    2531 scope.go:117] "RemoveContainer" containerID="d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"
Jul 27 20:14:24.348980 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:24.348962    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c\": container with ID starting with d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c not found: ID does not exist" containerID="d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"
Jul 27 20:14:24.349027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.348988    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c"} err="failed to get container status \"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c\": rpc error: code = NotFound desc = could not find container \"d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c\": container with ID starting with d139159810ea781b2f896af6c96203e0de65c5a83eb9c70c54d859786ec74d8c not found: ID does not exist"
Jul 27 20:14:24.365246 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.365222    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:14:24.374918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:24.374892    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ddrws"]
Jul 27 20:14:26.339003 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:26.338973    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" path="/var/lib/kubelet/pods/7cb1c55c-561f-42a9-89ee-45765a7c4dc5/volumes"
Jul 27 20:14:27.677322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.677287    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"]
Jul 27 20:14:27.677923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.677908    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" containerName="manager"
Jul 27 20:14:27.677985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.677925    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" containerName="manager"
Jul 27 20:14:27.678019 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.678008    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb1c55c-561f-42a9-89ee-45765a7c4dc5" containerName="manager"
Jul 27 20:14:27.682861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.682838    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.685150 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.685125    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qdshh\""
Jul 27 20:14:27.692481 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.692454    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"]
Jul 27 20:14:27.716881 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.716847    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.716881 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.716884    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.716905    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717025    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717048    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlm9\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-kube-api-access-jxlm9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717106    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717140    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717162    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.717415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.717200    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818014    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818055    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818072    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818093    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818266    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818291    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818318    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818375    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818401    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlm9\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-kube-api-access-jxlm9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818440    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818574 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818490    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818738    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.818861 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.818841    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.819116 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.819101    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.820431 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.820409    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.820679 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.820663    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.825953 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.825930    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.826075 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.826058    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlm9\" (UniqueName: \"kubernetes.io/projected/88a46126-6fe1-49e3-8ad3-076099d7b9d5-kube-api-access-jxlm9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-thjbb\" (UID: \"88a46126-6fe1-49e3-8ad3-076099d7b9d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:27.994731 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:27.994622    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:28.139122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.139013    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"]
Jul 27 20:14:28.142099 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:14:28.142064    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a46126_6fe1_49e3_8ad3_076099d7b9d5.slice/crio-d13e6224005be220bf747b85ee74e834fdd753c0dc0c815fa471f624d64347ef WatchSource:0}: Error finding container d13e6224005be220bf747b85ee74e834fdd753c0dc0c815fa471f624d64347ef: Status 404 returned error can't find the container with id d13e6224005be220bf747b85ee74e834fdd753c0dc0c815fa471f624d64347ef
Jul 27 20:14:28.144229 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.144191    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:14:28.144307 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.144265    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:14:28.144307 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.144299    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:14:28.355744 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.355705    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" event={"ID":"88a46126-6fe1-49e3-8ad3-076099d7b9d5","Type":"ContainerStarted","Data":"e735a7edce77f9b9fb6c536e79429f0a30089d823b973852b0e43fba314a5701"}
Jul 27 20:14:28.355935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.355768    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" event={"ID":"88a46126-6fe1-49e3-8ad3-076099d7b9d5","Type":"ContainerStarted","Data":"d13e6224005be220bf747b85ee74e834fdd753c0dc0c815fa471f624d64347ef"}
Jul 27 20:14:28.378777 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.378723    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" podStartSLOduration=1.378708487 podStartE2EDuration="1.378708487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:28.376103815 +0000 UTC m=+634.470689280" watchObservedRunningTime="2026-07-27 20:14:28.378708487 +0000 UTC m=+634.473293951"
Jul 27 20:14:28.995358 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:28.995323    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:29.999156 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:29.999128    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:30.365882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:30.365854    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:30.366821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:30.366805    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:14:41.512245 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.512203    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:41.517081 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.517064    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.519439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.519418    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 20:14:41.519576 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.519557    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f9gbg\""
Jul 27 20:14:41.526269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.526239    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:41.605454 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.605409    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:41.646382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.646351    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.646540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.646474    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rll5c\" (UniqueName: \"kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.747581 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.747545    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.747742 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.747647    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rll5c\" (UniqueName: \"kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.748276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.748257    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.755504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.755480    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rll5c\" (UniqueName: \"kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c\") pod \"limitador-limitador-64c94fd64f-5mrm7\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.828324 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.828287    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:41.971695 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:41.971629    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:41.974295 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:14:41.974258    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cacf75d_ce51_47b1_8aab_b0b9b4b53cd1.slice/crio-64f0e90b9992dbf8885d78c2abe76cc6d29ee5970068697b31b8ea52904c338b WatchSource:0}: Error finding container 64f0e90b9992dbf8885d78c2abe76cc6d29ee5970068697b31b8ea52904c338b: Status 404 returned error can't find the container with id 64f0e90b9992dbf8885d78c2abe76cc6d29ee5970068697b31b8ea52904c338b
Jul 27 20:14:42.426741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:42.426707    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" event={"ID":"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1","Type":"ContainerStarted","Data":"64f0e90b9992dbf8885d78c2abe76cc6d29ee5970068697b31b8ea52904c338b"}
Jul 27 20:14:45.446783 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:45.446746    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" event={"ID":"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1","Type":"ContainerStarted","Data":"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"}
Jul 27 20:14:45.447206 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:45.446840    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:45.466531 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:45.466478    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" podStartSLOduration=1.672225798 podStartE2EDuration="4.46646265s" totalImagesPullingTime="2.794236852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:45.464926215 +0000 UTC m=+651.559511673" watchObservedRunningTime="2026-07-27 20:14:45.46646265 +0000 UTC m=+651.561048113"
Jul 27 20:14:56.453052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:56.453022    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:58.920021 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:58.919983    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:58.920607 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:58.920227    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" podUID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" containerName="limitador" containerID="cri-o://39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e" gracePeriod=30
Jul 27 20:14:59.465364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.465336    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:59.520106 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.520078    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file\") pod \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") "
Jul 27 20:14:59.520288 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.520152    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rll5c\" (UniqueName: \"kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c\") pod \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\" (UID: \"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1\") "
Jul 27 20:14:59.520444 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.520421    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file" (OuterVolumeSpecName: "config-file") pod "5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" (UID: "5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:14:59.520596 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.520579    2531 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-config-file\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:14:59.522280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522252    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c" (OuterVolumeSpecName: "kube-api-access-rll5c") pod "5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" (UID: "5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1"). InnerVolumeSpecName "kube-api-access-rll5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:59.522730 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522709    2531 generic.go:356] "Generic (PLEG): container finished" podID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" containerID="39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e" exitCode=0
Jul 27 20:14:59.522793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522747    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"
Jul 27 20:14:59.522793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522775    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" event={"ID":"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1","Type":"ContainerDied","Data":"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"}
Jul 27 20:14:59.522856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522822    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5mrm7" event={"ID":"5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1","Type":"ContainerDied","Data":"64f0e90b9992dbf8885d78c2abe76cc6d29ee5970068697b31b8ea52904c338b"}
Jul 27 20:14:59.522856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.522843    2531 scope.go:117] "RemoveContainer" containerID="39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"
Jul 27 20:14:59.534944 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.534927    2531 scope.go:117] "RemoveContainer" containerID="39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"
Jul 27 20:14:59.535228 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:14:59.535209    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e\": container with ID starting with 39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e not found: ID does not exist" containerID="39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"
Jul 27 20:14:59.535315 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.535232    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e"} err="failed to get container status \"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e\": rpc error: code = NotFound desc = could not find container \"39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e\": container with ID starting with 39c3c41cb5bf3a2b55ac1cb1ccc9c14521e910ad5f9e6736422d612c813f2f9e not found: ID does not exist"
Jul 27 20:14:59.546186 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.546152    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:59.549837 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.549819    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5mrm7"]
Jul 27 20:14:59.621738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:14:59.621708    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rll5c\" (UniqueName: \"kubernetes.io/projected/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1-kube-api-access-rll5c\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:15:00.338737 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:00.338706    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" path="/var/lib/kubelet/pods/5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1/volumes"
Jul 27 20:15:21.832532 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.832475    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:21.832990 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.832975    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" containerName="limitador"
Jul 27 20:15:21.833044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.832994    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" containerName="limitador"
Jul 27 20:15:21.833079 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.833070    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cacf75d-ce51-47b1-8aab-b0b9b4b53cd1" containerName="limitador"
Jul 27 20:15:21.836537 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.836513    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:21.839055 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.839029    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qfxts\""
Jul 27 20:15:21.839207 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.839059    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 20:15:21.839207 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.839031    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 20:15:21.851798 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.851766    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:21.927842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.927800    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:21.927842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:21.927842    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9t9h\" (UniqueName: \"kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.009535 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.009492    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:15:22.013642 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.013619    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.024007 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.023981    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:15:22.029050 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.029025    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.029184 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.029057    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9t9h\" (UniqueName: \"kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.032155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.032129    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.038130 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.038105    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9t9h\" (UniqueName: \"kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h\") pod \"maas-controller-7f464d968f-br4bl\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") " pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.129730 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.129635    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.129881 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.129728    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprp6\" (UniqueName: \"kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.154021 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.153981    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:22.231292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.231252    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.231469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.231321    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprp6\" (UniqueName: \"kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.233802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.233761    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.240269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.240245    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprp6\" (UniqueName: \"kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6\") pod \"maas-controller-6c94868495-5wplk\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") " pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.295441 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.295409    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:22.296619 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:15:22.296592    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f3e240_5a0d_4221_aab8_4caba551a934.slice/crio-b18db05e48debbc931d21c0d1c927026dd6e9578fe0f57b94ac44ca907115b76 WatchSource:0}: Error finding container b18db05e48debbc931d21c0d1c927026dd6e9578fe0f57b94ac44ca907115b76: Status 404 returned error can't find the container with id b18db05e48debbc931d21c0d1c927026dd6e9578fe0f57b94ac44ca907115b76
Jul 27 20:15:22.324147 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.324116    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:22.475755 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.475718    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:15:22.479829 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:15:22.479799    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad718f4_caad_42ee_9213_9c3c6875bebf.slice/crio-56ee709cad749d4cc49547bfd85909ddc5d4d136329dc2847c615a6658cdc901 WatchSource:0}: Error finding container 56ee709cad749d4cc49547bfd85909ddc5d4d136329dc2847c615a6658cdc901: Status 404 returned error can't find the container with id 56ee709cad749d4cc49547bfd85909ddc5d4d136329dc2847c615a6658cdc901
Jul 27 20:15:22.644519 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.644466    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f464d968f-br4bl" event={"ID":"05f3e240-5a0d-4221-aab8-4caba551a934","Type":"ContainerStarted","Data":"b18db05e48debbc931d21c0d1c927026dd6e9578fe0f57b94ac44ca907115b76"}
Jul 27 20:15:22.645651 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:22.645626    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-5wplk" event={"ID":"2ad718f4-caad-42ee-9213-9c3c6875bebf","Type":"ContainerStarted","Data":"56ee709cad749d4cc49547bfd85909ddc5d4d136329dc2847c615a6658cdc901"}
Jul 27 20:15:25.674846 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.674806    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f464d968f-br4bl" event={"ID":"05f3e240-5a0d-4221-aab8-4caba551a934","Type":"ContainerStarted","Data":"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"}
Jul 27 20:15:25.675445 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.674989    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:25.676396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.676374    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-5wplk" event={"ID":"2ad718f4-caad-42ee-9213-9c3c6875bebf","Type":"ContainerStarted","Data":"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"}
Jul 27 20:15:25.676491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.676454    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:25.712046 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.712000    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f464d968f-br4bl" podStartSLOduration=2.106067003 podStartE2EDuration="4.711985701s" totalImagesPullingTime="2.605918698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:25.710219166 +0000 UTC m=+691.804804630" watchObservedRunningTime="2026-07-27 20:15:25.711985701 +0000 UTC m=+691.806571164"
Jul 27 20:15:25.758729 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:25.758665    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c94868495-5wplk" podStartSLOduration=2.325302466 podStartE2EDuration="4.758651902s" totalImagesPullingTime="2.433349436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:25.747466217 +0000 UTC m=+691.842051681" watchObservedRunningTime="2026-07-27 20:15:25.758651902 +0000 UTC m=+691.853237366"
Jul 27 20:15:36.683096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:36.683066    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:15:36.683480 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:36.683133    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:36.749954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:36.749917    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:36.750196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:36.750151    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f464d968f-br4bl" podUID="05f3e240-5a0d-4221-aab8-4caba551a934" containerName="manager" containerID="cri-o://cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42" gracePeriod=30
Jul 27 20:15:37.001470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.001442    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:37.179706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.179656    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9t9h\" (UniqueName: \"kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h\") pod \"05f3e240-5a0d-4221-aab8-4caba551a934\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") "
Jul 27 20:15:37.179896 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.179720    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert\") pod \"05f3e240-5a0d-4221-aab8-4caba551a934\" (UID: \"05f3e240-5a0d-4221-aab8-4caba551a934\") "
Jul 27 20:15:37.182168 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.182136    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h" (OuterVolumeSpecName: "kube-api-access-k9t9h") pod "05f3e240-5a0d-4221-aab8-4caba551a934" (UID: "05f3e240-5a0d-4221-aab8-4caba551a934"). InnerVolumeSpecName "kube-api-access-k9t9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:15:37.182302 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.182162    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "05f3e240-5a0d-4221-aab8-4caba551a934" (UID: "05f3e240-5a0d-4221-aab8-4caba551a934"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:15:37.281778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.281750    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9t9h\" (UniqueName: \"kubernetes.io/projected/05f3e240-5a0d-4221-aab8-4caba551a934-kube-api-access-k9t9h\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:15:37.281778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.281778    2531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f3e240-5a0d-4221-aab8-4caba551a934-webhook-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:15:37.738786 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.738698    2531 generic.go:356] "Generic (PLEG): container finished" podID="05f3e240-5a0d-4221-aab8-4caba551a934" containerID="cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42" exitCode=0
Jul 27 20:15:37.738786 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.738741    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f464d968f-br4bl" event={"ID":"05f3e240-5a0d-4221-aab8-4caba551a934","Type":"ContainerDied","Data":"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"}
Jul 27 20:15:37.738786 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.738761    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f464d968f-br4bl"
Jul 27 20:15:37.739298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.738792    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f464d968f-br4bl" event={"ID":"05f3e240-5a0d-4221-aab8-4caba551a934","Type":"ContainerDied","Data":"b18db05e48debbc931d21c0d1c927026dd6e9578fe0f57b94ac44ca907115b76"}
Jul 27 20:15:37.739298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.738816    2531 scope.go:117] "RemoveContainer" containerID="cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"
Jul 27 20:15:37.747921 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.747904    2531 scope.go:117] "RemoveContainer" containerID="cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"
Jul 27 20:15:37.748157 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:15:37.748138    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42\": container with ID starting with cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42 not found: ID does not exist" containerID="cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"
Jul 27 20:15:37.748260 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.748168    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42"} err="failed to get container status \"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42\": rpc error: code = NotFound desc = could not find container \"cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42\": container with ID starting with cf24c607b82118dd703eea9b2e917fec914527b5d89541cb6439976126035a42 not found: ID does not exist"
Jul 27 20:15:37.762232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.762204    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:37.765705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:37.765685    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f464d968f-br4bl"]
Jul 27 20:15:38.339473 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:38.339444    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f3e240-5a0d-4221-aab8-4caba551a934" path="/var/lib/kubelet/pods/05f3e240-5a0d-4221-aab8-4caba551a934/volumes"
Jul 27 20:15:58.571624 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.571589    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"]
Jul 27 20:15:58.572029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.572014    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f3e240-5a0d-4221-aab8-4caba551a934" containerName="manager"
Jul 27 20:15:58.572029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.572030    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f3e240-5a0d-4221-aab8-4caba551a934" containerName="manager"
Jul 27 20:15:58.572120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.572111    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f3e240-5a0d-4221-aab8-4caba551a934" containerName="manager"
Jul 27 20:15:58.581027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.581003    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.583606 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.583582    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-flzfl\""
Jul 27 20:15:58.583745 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.583645    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 20:15:58.584012 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.583996    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 20:15:58.584134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.584119    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 20:15:58.589979 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.589961    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"]
Jul 27 20:15:58.697927 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.697894    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mdm9\" (UniqueName: \"kubernetes.io/projected/769523bf-f746-4e90-a6e7-a92cb240246b-kube-api-access-4mdm9\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.697927 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.697929    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769523bf-f746-4e90-a6e7-a92cb240246b-openshift-service-ca\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.698137 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.697962    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769523bf-f746-4e90-a6e7-a92cb240246b-maas-api-tls\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.799353 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.799314    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769523bf-f746-4e90-a6e7-a92cb240246b-openshift-service-ca\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.799538 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.799388    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769523bf-f746-4e90-a6e7-a92cb240246b-maas-api-tls\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.799538 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.799525    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mdm9\" (UniqueName: \"kubernetes.io/projected/769523bf-f746-4e90-a6e7-a92cb240246b-kube-api-access-4mdm9\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.799979 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.799957    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769523bf-f746-4e90-a6e7-a92cb240246b-openshift-service-ca\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.801842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.801819    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769523bf-f746-4e90-a6e7-a92cb240246b-maas-api-tls\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.808469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.808442    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mdm9\" (UniqueName: \"kubernetes.io/projected/769523bf-f746-4e90-a6e7-a92cb240246b-kube-api-access-4mdm9\") pod \"maas-api-848d79ddd8-46d9r\" (UID: \"769523bf-f746-4e90-a6e7-a92cb240246b\") " pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:58.892821 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:58.892737    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:15:59.045705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.045671    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"]
Jul 27 20:15:59.193402 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.193324    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"]
Jul 27 20:15:59.198992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.198970    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.201992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.201970    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vgmmk\""
Jul 27 20:15:59.202431 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.202415    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 20:15:59.206590 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.206301    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"]
Jul 27 20:15:59.287793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.287754    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-6vxxd"]
Jul 27 20:15:59.291913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.291886    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.303439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.303399    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9r5j\" (UniqueName: \"kubernetes.io/projected/32f3847f-0bdb-45ea-b166-2e3acd32619e-kube-api-access-x9r5j\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.303769 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.303747    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f3847f-0bdb-45ea-b166-2e3acd32619e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.305015 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.304988    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-6vxxd"]
Jul 27 20:15:59.404441 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.404409    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f3847f-0bdb-45ea-b166-2e3acd32619e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.404651 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.404455    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9r5j\" (UniqueName: \"kubernetes.io/projected/32f3847f-0bdb-45ea-b166-2e3acd32619e-kube-api-access-x9r5j\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.404651 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.404510    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9b250e-8139-4ef7-8128-cc1c7c228ee6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.404651 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.404530    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t42l\" (UniqueName: \"kubernetes.io/projected/da9b250e-8139-4ef7-8128-cc1c7c228ee6-kube-api-access-8t42l\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.405256 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.405234    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f3847f-0bdb-45ea-b166-2e3acd32619e-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.414809 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.414784    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9r5j\" (UniqueName: \"kubernetes.io/projected/32f3847f-0bdb-45ea-b166-2e3acd32619e-kube-api-access-x9r5j\") pod \"payload-pre-processing-687dbb6cf6-hwgww\" (UID: \"32f3847f-0bdb-45ea-b166-2e3acd32619e\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.505423 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.505338    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t42l\" (UniqueName: \"kubernetes.io/projected/da9b250e-8139-4ef7-8128-cc1c7c228ee6-kube-api-access-8t42l\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.505638 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.505460    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9b250e-8139-4ef7-8128-cc1c7c228ee6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.506101 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.506083    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9b250e-8139-4ef7-8128-cc1c7c228ee6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.513191 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.513153    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:15:59.523822 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.523797    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t42l\" (UniqueName: \"kubernetes.io/projected/da9b250e-8139-4ef7-8128-cc1c7c228ee6-kube-api-access-8t42l\") pod \"payload-processing-5cc6cd444b-6vxxd\" (UID: \"da9b250e-8139-4ef7-8128-cc1c7c228ee6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.604564 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.604520    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:15:59.683898 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.683870    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"]
Jul 27 20:15:59.685894 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:15:59.685865    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f3847f_0bdb_45ea_b166_2e3acd32619e.slice/crio-1c80b5a237a046b1e0a7106fc95e9070c6467e7fe203ff593eae8d7f3eef22df WatchSource:0}: Error finding container 1c80b5a237a046b1e0a7106fc95e9070c6467e7fe203ff593eae8d7f3eef22df: Status 404 returned error can't find the container with id 1c80b5a237a046b1e0a7106fc95e9070c6467e7fe203ff593eae8d7f3eef22df
Jul 27 20:15:59.855196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.854480    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-6vxxd"]
Jul 27 20:15:59.876412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.866727    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r" event={"ID":"769523bf-f746-4e90-a6e7-a92cb240246b","Type":"ContainerStarted","Data":"be1799fe43b5407347fe265a9f0b5fb0c1468522dc78e8d654be932d8cfdd2b6"}
Jul 27 20:15:59.876412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:15:59.868330    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww" event={"ID":"32f3847f-0bdb-45ea-b166-2e3acd32619e","Type":"ContainerStarted","Data":"1c80b5a237a046b1e0a7106fc95e9070c6467e7fe203ff593eae8d7f3eef22df"}
Jul 27 20:15:59.876412 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:15:59.876262    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9b250e_8139_4ef7_8128_cc1c7c228ee6.slice/crio-7135217af204eca12b26250f9d209b7bc0537d698eaad428897f59aaa03cfaf3 WatchSource:0}: Error finding container 7135217af204eca12b26250f9d209b7bc0537d698eaad428897f59aaa03cfaf3: Status 404 returned error can't find the container with id 7135217af204eca12b26250f9d209b7bc0537d698eaad428897f59aaa03cfaf3
Jul 27 20:16:00.883961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:00.883917    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd" event={"ID":"da9b250e-8139-4ef7-8128-cc1c7c228ee6","Type":"ContainerStarted","Data":"7135217af204eca12b26250f9d209b7bc0537d698eaad428897f59aaa03cfaf3"}
Jul 27 20:16:01.891584 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:01.891550    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r" event={"ID":"769523bf-f746-4e90-a6e7-a92cb240246b","Type":"ContainerStarted","Data":"25094461103a8f8800813df13073bdc649627db1eb151d17c5c12a3f52a77f05"}
Jul 27 20:16:01.891995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:01.891663    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:16:01.911374 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:01.911300    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r" podStartSLOduration=1.734046067 podStartE2EDuration="3.91128102s" totalImagesPullingTime="2.177234953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:16:01.908282014 +0000 UTC m=+728.002867493" watchObservedRunningTime="2026-07-27 20:16:01.91128102 +0000 UTC m=+728.005866485"
Jul 27 20:16:03.906487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.906449    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww" event={"ID":"32f3847f-0bdb-45ea-b166-2e3acd32619e","Type":"ContainerStarted","Data":"21edd5af3521438d19fc972f1bcccc93f9ebc7aa5e6424b135a322b2e4394f4e"}
Jul 27 20:16:03.906948 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.906537    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:16:03.908039 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.908014    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd" event={"ID":"da9b250e-8139-4ef7-8128-cc1c7c228ee6","Type":"ContainerStarted","Data":"55c2006076ee9477458afee7fec11f0f4463de1c64e2ac3abf7a357e9048c1d6"}
Jul 27 20:16:03.908207 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.908153    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:16:03.925836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.925791    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww" podStartSLOduration=1.530109501 podStartE2EDuration="4.925776293s" totalImagesPullingTime="3.395666792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:16:03.923771268 +0000 UTC m=+730.018356756" watchObservedRunningTime="2026-07-27 20:16:03.925776293 +0000 UTC m=+730.020361757"
Jul 27 20:16:03.941314 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:03.941255    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd" podStartSLOduration=1.742356735 podStartE2EDuration="4.941234296s" totalImagesPullingTime="3.198877561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:16:03.94038195 +0000 UTC m=+730.034967417" watchObservedRunningTime="2026-07-27 20:16:03.941234296 +0000 UTC m=+730.035819763"
Jul 27 20:16:06.925765 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.925729    2531 generic.go:356] "Generic (PLEG): container finished" podID="88a46126-6fe1-49e3-8ad3-076099d7b9d5" containerID="e735a7edce77f9b9fb6c536e79429f0a30089d823b973852b0e43fba314a5701" exitCode=0
Jul 27 20:16:06.926266 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.925805    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" event={"ID":"88a46126-6fe1-49e3-8ad3-076099d7b9d5","Type":"ContainerDied","Data":"e735a7edce77f9b9fb6c536e79429f0a30089d823b973852b0e43fba314a5701"}
Jul 27 20:16:06.926341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.926273    2531 scope.go:117] "RemoveContainer" containerID="e735a7edce77f9b9fb6c536e79429f0a30089d823b973852b0e43fba314a5701"
Jul 27 20:16:06.927280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.927256    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:16:06.927338 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.927319    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:16:06.927376 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:06.927352    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:16:07.905246 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:07.905212    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-848d79ddd8-46d9r"
Jul 27 20:16:07.936300 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:07.936265    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" event={"ID":"88a46126-6fe1-49e3-8ad3-076099d7b9d5","Type":"ContainerStarted","Data":"ab699531ce32dca71655eefb8c1c9d73d9a96bee42dc8465eef217c000e36e91"}
Jul 27 20:16:07.994941 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:07.994908    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:16:07.995095 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:07.995069    2531 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 27 20:16:07.995155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:07.995114    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" podUID="88a46126-6fe1-49e3-8ad3-076099d7b9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 27 20:16:08.995349 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:08.995316    2531 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 27 20:16:08.995739 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:08.995383    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" podUID="88a46126-6fe1-49e3-8ad3-076099d7b9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 27 20:16:09.994961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:09.994928    2531 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 27 20:16:09.995113 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:09.994983    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" podUID="88a46126-6fe1-49e3-8ad3-076099d7b9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 27 20:16:10.995696 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:10.995659    2531 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-thjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 27 20:16:10.996068 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:10.995715    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb" podUID="88a46126-6fe1-49e3-8ad3-076099d7b9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 27 20:16:11.998306 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:11.998279    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:16:11.998693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:11.998520    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:16:11.999192 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:11.999156    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-thjbb"
Jul 27 20:16:14.915518 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:14.915491    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hwgww"
Jul 27 20:16:14.915979 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:16:14.915570    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-6vxxd"
Jul 27 20:17:21.493262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.493140    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"]
Jul 27 20:17:21.497394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.497365    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.500909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.500880    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qvwsc\""
Jul 27 20:17:21.500909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.500896    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 20:17:21.501149 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.500936    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 20:17:21.501149 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.501040    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 20:17:21.504441 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.504409    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"]
Jul 27 20:17:21.654867 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.654825    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmpv\" (UniqueName: \"kubernetes.io/projected/1eab6677-9803-4865-a8cb-0ab2be2633dc-kube-api-access-sdmpv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.655061 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.654991    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.655061 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.655039    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.655155 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.655096    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eab6677-9803-4865-a8cb-0ab2be2633dc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.655234 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.655193    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.655295 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.655279    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.755935    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.755980    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.755999    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eab6677-9803-4865-a8cb-0ab2be2633dc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756025    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756094    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756168    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmpv\" (UniqueName: \"kubernetes.io/projected/1eab6677-9803-4865-a8cb-0ab2be2633dc-kube-api-access-sdmpv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756415    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756451    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.756516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.756505    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.758571 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.758548    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eab6677-9803-4865-a8cb-0ab2be2633dc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.758747 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.758731    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eab6677-9803-4865-a8cb-0ab2be2633dc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.764144 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.764117    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmpv\" (UniqueName: \"kubernetes.io/projected/1eab6677-9803-4865-a8cb-0ab2be2633dc-kube-api-access-sdmpv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4hx7r\" (UID: \"1eab6677-9803-4865-a8cb-0ab2be2633dc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.813033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.812990    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:21.953041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.953010    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"]
Jul 27 20:17:21.954637 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:17:21.954609    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eab6677_9803_4865_a8cb_0ab2be2633dc.slice/crio-4e4ada9324ff2cc8f1f0e9bd1252cbb0a4fa2e155c6267961ffd04380b4d68f6 WatchSource:0}: Error finding container 4e4ada9324ff2cc8f1f0e9bd1252cbb0a4fa2e155c6267961ffd04380b4d68f6: Status 404 returned error can't find the container with id 4e4ada9324ff2cc8f1f0e9bd1252cbb0a4fa2e155c6267961ffd04380b4d68f6
Jul 27 20:17:21.956209 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:21.956161    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:17:22.354849 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:22.354809    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r" event={"ID":"1eab6677-9803-4865-a8cb-0ab2be2633dc","Type":"ContainerStarted","Data":"4e4ada9324ff2cc8f1f0e9bd1252cbb0a4fa2e155c6267961ffd04380b4d68f6"}
Jul 27 20:17:23.361796 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:23.361707    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r" event={"ID":"1eab6677-9803-4865-a8cb-0ab2be2633dc","Type":"ContainerStarted","Data":"e346b3a2ec70047d755b99e2e3c86ddc452ba3987c57720827f9563403806960"}
Jul 27 20:17:23.399878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:23.399822    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r" podStartSLOduration=1.322404494 podStartE2EDuration="2.399804883s" totalImagesPullingTime="1.077400389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:23.394604976 +0000 UTC m=+809.489190441" watchObservedRunningTime="2026-07-27 20:17:23.399804883 +0000 UTC m=+809.494390347"
Jul 27 20:17:31.813316 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:31.813272    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:31.813814 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:31.813342    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:31.826147 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:31.826117    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:32.425167 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:32.425131    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4hx7r"
Jul 27 20:17:41.692331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.692290    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"]
Jul 27 20:17:41.696685 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.696659    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.699237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.699214    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 27 20:17:41.704564 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.704222    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"]
Jul 27 20:17:41.755319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755281    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.755516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755359    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.755516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755418    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.755516 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755482    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.755648 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755518    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fxt\" (UniqueName: \"kubernetes.io/projected/d61208ed-71a2-4af4-a0c5-d57efbc5097f-kube-api-access-b8fxt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.755648 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.755550    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856488    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856535    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856557    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fxt\" (UniqueName: \"kubernetes.io/projected/d61208ed-71a2-4af4-a0c5-d57efbc5097f-kube-api-access-b8fxt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856598    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856776    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856897    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.856951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856921    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.857045 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.856976    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.857224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.857169    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.859002 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.858966    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61208ed-71a2-4af4-a0c5-d57efbc5097f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.859220 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.859203    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61208ed-71a2-4af4-a0c5-d57efbc5097f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:41.864450 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:41.864426    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fxt\" (UniqueName: \"kubernetes.io/projected/d61208ed-71a2-4af4-a0c5-d57efbc5097f-kube-api-access-b8fxt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qzfsm\" (UID: \"d61208ed-71a2-4af4-a0c5-d57efbc5097f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:42.008584 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:42.008495    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:42.152817 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:42.152778    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"]
Jul 27 20:17:42.154459 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:17:42.154422    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61208ed_71a2_4af4_a0c5_d57efbc5097f.slice/crio-30534436fe5bd93fb15355fe723ca97c48e326802a869580491547bc913d817e WatchSource:0}: Error finding container 30534436fe5bd93fb15355fe723ca97c48e326802a869580491547bc913d817e: Status 404 returned error can't find the container with id 30534436fe5bd93fb15355fe723ca97c48e326802a869580491547bc913d817e
Jul 27 20:17:42.473626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:42.473596    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm" event={"ID":"d61208ed-71a2-4af4-a0c5-d57efbc5097f","Type":"ContainerStarted","Data":"30534436fe5bd93fb15355fe723ca97c48e326802a869580491547bc913d817e"}
Jul 27 20:17:43.480331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:43.480288    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm" event={"ID":"d61208ed-71a2-4af4-a0c5-d57efbc5097f","Type":"ContainerStarted","Data":"2ab92bc6e1dc492425ed1c94047dacabff558a8aed6a9e3400f51af1c6f0b073"}
Jul 27 20:17:43.502601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:43.502542    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm" podStartSLOduration=2.19045628 podStartE2EDuration="2.502523514s" totalImagesPullingTime="312.067234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:43.501010187 +0000 UTC m=+829.595595651" watchObservedRunningTime="2026-07-27 20:17:43.502523514 +0000 UTC m=+829.597108977"
Jul 27 20:17:52.008966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:52.008925    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:52.009441 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:52.009005    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:52.021716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:52.021682    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:17:52.548778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:17:52.548743    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qzfsm"
Jul 27 20:18:54.333331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:18:54.333244    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:18:54.334833 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:18:54.334200    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:19:37.397956 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.397916    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:19:37.398517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.398155    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c94868495-5wplk" podUID="2ad718f4-caad-42ee-9213-9c3c6875bebf" containerName="manager" containerID="cri-o://eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453" gracePeriod=30
Jul 27 20:19:37.665415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.665339    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:19:37.717948 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.717909    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert\") pod \"2ad718f4-caad-42ee-9213-9c3c6875bebf\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") "
Jul 27 20:19:37.718135 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.718005    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dprp6\" (UniqueName: \"kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6\") pod \"2ad718f4-caad-42ee-9213-9c3c6875bebf\" (UID: \"2ad718f4-caad-42ee-9213-9c3c6875bebf\") "
Jul 27 20:19:37.720134 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.720092    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2ad718f4-caad-42ee-9213-9c3c6875bebf" (UID: "2ad718f4-caad-42ee-9213-9c3c6875bebf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:19:37.720300 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.720157    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6" (OuterVolumeSpecName: "kube-api-access-dprp6") pod "2ad718f4-caad-42ee-9213-9c3c6875bebf" (UID: "2ad718f4-caad-42ee-9213-9c3c6875bebf"). InnerVolumeSpecName "kube-api-access-dprp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:19:37.819491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.819447    2531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2ad718f4-caad-42ee-9213-9c3c6875bebf-webhook-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:19:37.819491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:37.819488    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprp6\" (UniqueName: \"kubernetes.io/projected/2ad718f4-caad-42ee-9213-9c3c6875bebf-kube-api-access-dprp6\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:19:38.114383 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.114345    2531 generic.go:356] "Generic (PLEG): container finished" podID="2ad718f4-caad-42ee-9213-9c3c6875bebf" containerID="eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453" exitCode=0
Jul 27 20:19:38.114589 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.114389    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-5wplk"
Jul 27 20:19:38.114589 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.114428    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-5wplk" event={"ID":"2ad718f4-caad-42ee-9213-9c3c6875bebf","Type":"ContainerDied","Data":"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"}
Jul 27 20:19:38.114589 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.114481    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-5wplk" event={"ID":"2ad718f4-caad-42ee-9213-9c3c6875bebf","Type":"ContainerDied","Data":"56ee709cad749d4cc49547bfd85909ddc5d4d136329dc2847c615a6658cdc901"}
Jul 27 20:19:38.114589 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.114504    2531 scope.go:117] "RemoveContainer" containerID="eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"
Jul 27 20:19:38.123838 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.123821    2531 scope.go:117] "RemoveContainer" containerID="eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"
Jul 27 20:19:38.124097 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:19:38.124077    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453\": container with ID starting with eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453 not found: ID does not exist" containerID="eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"
Jul 27 20:19:38.124167 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.124105    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453"} err="failed to get container status \"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453\": rpc error: code = NotFound desc = could not find container \"eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453\": container with ID starting with eab8beaca6f30180f7241a5a47af5fa7472990ce24fd67c8153d1bfbbdd40453 not found: ID does not exist"
Jul 27 20:19:38.148548 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.148521    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:19:38.160682 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.160658    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c94868495-5wplk"]
Jul 27 20:19:38.340026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:19:38.339990    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad718f4-caad-42ee-9213-9c3c6875bebf" path="/var/lib/kubelet/pods/2ad718f4-caad-42ee-9213-9c3c6875bebf/volumes"
Jul 27 20:20:38.983122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.983075    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:38.983796 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.983774    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad718f4-caad-42ee-9213-9c3c6875bebf" containerName="manager"
Jul 27 20:20:38.983889 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.983799    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad718f4-caad-42ee-9213-9c3c6875bebf" containerName="manager"
Jul 27 20:20:38.983953 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.983931    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad718f4-caad-42ee-9213-9c3c6875bebf" containerName="manager"
Jul 27 20:20:38.987189 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.987152    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:38.990362 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.990338    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 20:20:38.990504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.990338    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 20:20:38.990504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.990343    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qfxts\""
Jul 27 20:20:38.998936 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:38.998909    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:39.035911 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.035872    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.036094 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.035926    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmr4\" (UniqueName: \"kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.137652 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.137615    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.137877 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.137666    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmr4\" (UniqueName: \"kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.140149 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.140124    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.145371 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.145347    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmr4\" (UniqueName: \"kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4\") pod \"maas-controller-6c94868495-b5vx6\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") " pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.297764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.297729    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:39.437806 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.437769    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:39.440567 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:20:39.440532    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c6e481_003f_43b0_a8fb_5551b98b56dd.slice/crio-68e691fe3026549c6cec18a85c1778000c75aed31e56ff7f453ed7c80be04fc1 WatchSource:0}: Error finding container 68e691fe3026549c6cec18a85c1778000c75aed31e56ff7f453ed7c80be04fc1: Status 404 returned error can't find the container with id 68e691fe3026549c6cec18a85c1778000c75aed31e56ff7f453ed7c80be04fc1
Jul 27 20:20:39.456676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:39.456645    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-b5vx6" event={"ID":"83c6e481-003f-43b0-a8fb-5551b98b56dd","Type":"ContainerStarted","Data":"68e691fe3026549c6cec18a85c1778000c75aed31e56ff7f453ed7c80be04fc1"}
Jul 27 20:20:40.464164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:40.464068    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-b5vx6" event={"ID":"83c6e481-003f-43b0-a8fb-5551b98b56dd","Type":"ContainerStarted","Data":"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"}
Jul 27 20:20:40.464584 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:40.464229    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:40.483526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:40.483477    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c94868495-b5vx6" podStartSLOduration=1.847156794 podStartE2EDuration="2.483462023s" totalImagesPullingTime="636.305229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:40.481240387 +0000 UTC m=+1006.575825851" watchObservedRunningTime="2026-07-27 20:20:40.483462023 +0000 UTC m=+1006.578047485"
Jul 27 20:20:51.471854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:51.471823    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:56.113271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.113228    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:56.113899 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.113469    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c94868495-b5vx6" podUID="83c6e481-003f-43b0-a8fb-5551b98b56dd" containerName="manager" containerID="cri-o://bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67" gracePeriod=30
Jul 27 20:20:56.367893 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.367833    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:56.518299 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.518252    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qmr4\" (UniqueName: \"kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4\") pod \"83c6e481-003f-43b0-a8fb-5551b98b56dd\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") "
Jul 27 20:20:56.518479 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.518333    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert\") pod \"83c6e481-003f-43b0-a8fb-5551b98b56dd\" (UID: \"83c6e481-003f-43b0-a8fb-5551b98b56dd\") "
Jul 27 20:20:56.520466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.520433    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4" (OuterVolumeSpecName: "kube-api-access-8qmr4") pod "83c6e481-003f-43b0-a8fb-5551b98b56dd" (UID: "83c6e481-003f-43b0-a8fb-5551b98b56dd"). InnerVolumeSpecName "kube-api-access-8qmr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:56.520466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.520432    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "83c6e481-003f-43b0-a8fb-5551b98b56dd" (UID: "83c6e481-003f-43b0-a8fb-5551b98b56dd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:56.555281 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.555245    2531 generic.go:356] "Generic (PLEG): container finished" podID="83c6e481-003f-43b0-a8fb-5551b98b56dd" containerID="bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67" exitCode=0
Jul 27 20:20:56.555439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.555289    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-b5vx6"
Jul 27 20:20:56.555439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.555332    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-b5vx6" event={"ID":"83c6e481-003f-43b0-a8fb-5551b98b56dd","Type":"ContainerDied","Data":"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"}
Jul 27 20:20:56.555439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.555389    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-b5vx6" event={"ID":"83c6e481-003f-43b0-a8fb-5551b98b56dd","Type":"ContainerDied","Data":"68e691fe3026549c6cec18a85c1778000c75aed31e56ff7f453ed7c80be04fc1"}
Jul 27 20:20:56.555439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.555417    2531 scope.go:117] "RemoveContainer" containerID="bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"
Jul 27 20:20:56.565031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.565015    2531 scope.go:117] "RemoveContainer" containerID="bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"
Jul 27 20:20:56.565282 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:20:56.565259    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67\": container with ID starting with bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67 not found: ID does not exist" containerID="bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"
Jul 27 20:20:56.565352 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.565294    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67"} err="failed to get container status \"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67\": rpc error: code = NotFound desc = could not find container \"bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67\": container with ID starting with bcfcb321a2c73908147c38a487a033a41c4397b26bf5a83662c25051f7b01f67 not found: ID does not exist"
Jul 27 20:20:56.578484 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.578462    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:56.582051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.582033    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c94868495-b5vx6"]
Jul 27 20:20:56.619583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.619529    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmr4\" (UniqueName: \"kubernetes.io/projected/83c6e481-003f-43b0-a8fb-5551b98b56dd-kube-api-access-8qmr4\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:20:56.619583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:56.619549    2531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83c6e481-003f-43b0-a8fb-5551b98b56dd-webhook-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:20:57.400381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.400349    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c94868495-fln88"]
Jul 27 20:20:57.400873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.400857    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c6e481-003f-43b0-a8fb-5551b98b56dd" containerName="manager"
Jul 27 20:20:57.400873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.400874    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c6e481-003f-43b0-a8fb-5551b98b56dd" containerName="manager"
Jul 27 20:20:57.400998 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.400968    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c6e481-003f-43b0-a8fb-5551b98b56dd" containerName="manager"
Jul 27 20:20:57.405469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.405438    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.408722 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.408697    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qfxts\""
Jul 27 20:20:57.408837 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.408757    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 20:20:57.408908 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.408840    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 20:20:57.415755 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.415730    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-fln88"]
Jul 27 20:20:57.529396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.529358    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcth4\" (UniqueName: \"kubernetes.io/projected/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-kube-api-access-bcth4\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.529570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.529429    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-webhook-cert\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.630080 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.630040    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcth4\" (UniqueName: \"kubernetes.io/projected/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-kube-api-access-bcth4\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.630276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.630138    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-webhook-cert\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.633102 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.633066    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-webhook-cert\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.643645 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.643613    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcth4\" (UniqueName: \"kubernetes.io/projected/b0902d67-74fa-4f5f-a44a-aaaae91e33d9-kube-api-access-bcth4\") pod \"maas-controller-6c94868495-fln88\" (UID: \"b0902d67-74fa-4f5f-a44a-aaaae91e33d9\") " pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.717761 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.717680    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:57.858015 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:57.857976    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c94868495-fln88"]
Jul 27 20:20:57.861727 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:20:57.861697    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0902d67_74fa_4f5f_a44a_aaaae91e33d9.slice/crio-0ee0933f208a23c4cf68e27400630347fb73e4d86c0ebb0485e5d68a1a14e78f WatchSource:0}: Error finding container 0ee0933f208a23c4cf68e27400630347fb73e4d86c0ebb0485e5d68a1a14e78f: Status 404 returned error can't find the container with id 0ee0933f208a23c4cf68e27400630347fb73e4d86c0ebb0485e5d68a1a14e78f
Jul 27 20:20:58.340689 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:58.340663    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c6e481-003f-43b0-a8fb-5551b98b56dd" path="/var/lib/kubelet/pods/83c6e481-003f-43b0-a8fb-5551b98b56dd/volumes"
Jul 27 20:20:58.571894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:58.571800    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-fln88" event={"ID":"b0902d67-74fa-4f5f-a44a-aaaae91e33d9","Type":"ContainerStarted","Data":"bd6a1060e7395f97c1a974556c5c16653295d78ff954f547145181137f47ad15"}
Jul 27 20:20:58.571894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:58.571856    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c94868495-fln88" event={"ID":"b0902d67-74fa-4f5f-a44a-aaaae91e33d9","Type":"ContainerStarted","Data":"0ee0933f208a23c4cf68e27400630347fb73e4d86c0ebb0485e5d68a1a14e78f"}
Jul 27 20:20:58.572326 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:58.571910    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:20:58.591381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:20:58.591332    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c94868495-fln88" podStartSLOduration=1.169706655 podStartE2EDuration="1.591318629s" totalImagesPullingTime="421.611974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:58.589194244 +0000 UTC m=+1024.683779703" watchObservedRunningTime="2026-07-27 20:20:58.591318629 +0000 UTC m=+1024.685904094"
Jul 27 20:21:09.579883 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:21:09.579852    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c94868495-fln88"
Jul 27 20:23:54.382704 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:23:54.382670    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:23:54.385650 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:23:54.385624    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:25:55.508938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:55.508895    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:25:55.509542 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:55.509109    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" podUID="09bba92c-0784-4133-a132-d1f335a63f9b" containerName="manager" containerID="cri-o://6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8" gracePeriod=10
Jul 27 20:25:58.059328 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.059301    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:25:58.164741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.164648    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume\") pod \"09bba92c-0784-4133-a132-d1f335a63f9b\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") "
Jul 27 20:25:58.164909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.164762    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpcnq\" (UniqueName: \"kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq\") pod \"09bba92c-0784-4133-a132-d1f335a63f9b\" (UID: \"09bba92c-0784-4133-a132-d1f335a63f9b\") "
Jul 27 20:25:58.165043 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.165018    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "09bba92c-0784-4133-a132-d1f335a63f9b" (UID: "09bba92c-0784-4133-a132-d1f335a63f9b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:25:58.165137 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.165120    2531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/09bba92c-0784-4133-a132-d1f335a63f9b-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:25:58.166854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.166830    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq" (OuterVolumeSpecName: "kube-api-access-tpcnq") pod "09bba92c-0784-4133-a132-d1f335a63f9b" (UID: "09bba92c-0784-4133-a132-d1f335a63f9b"). InnerVolumeSpecName "kube-api-access-tpcnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:25:58.220849 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.220817    2531 generic.go:356] "Generic (PLEG): container finished" podID="09bba92c-0784-4133-a132-d1f335a63f9b" containerID="6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8" exitCode=0
Jul 27 20:25:58.220994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.220865    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"
Jul 27 20:25:58.220994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.220907    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" event={"ID":"09bba92c-0784-4133-a132-d1f335a63f9b","Type":"ContainerDied","Data":"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"}
Jul 27 20:25:58.220994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.220962    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd" event={"ID":"09bba92c-0784-4133-a132-d1f335a63f9b","Type":"ContainerDied","Data":"dfc793081f4e6be171ac6a2550a1d86bd855126174eb2b10618aa383044c35d7"}
Jul 27 20:25:58.220994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.220993    2531 scope.go:117] "RemoveContainer" containerID="6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"
Jul 27 20:25:58.231344 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.231325    2531 scope.go:117] "RemoveContainer" containerID="6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"
Jul 27 20:25:58.231609 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:25:58.231592    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8\": container with ID starting with 6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8 not found: ID does not exist" containerID="6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"
Jul 27 20:25:58.231664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.231617    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8"} err="failed to get container status \"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8\": rpc error: code = NotFound desc = could not find container \"6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8\": container with ID starting with 6b4d059fdbcd7598e4972cbc08ee05f131a7e4bafc7087759b4865548f61c5b8 not found: ID does not exist"
Jul 27 20:25:58.243415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.243392    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:25:58.247744 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.247725    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wkbbd"]
Jul 27 20:25:58.266398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.266376    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcnq\" (UniqueName: \"kubernetes.io/projected/09bba92c-0784-4133-a132-d1f335a63f9b-kube-api-access-tpcnq\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:25:58.339288 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:25:58.339259    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09bba92c-0784-4133-a132-d1f335a63f9b" path="/var/lib/kubelet/pods/09bba92c-0784-4133-a132-d1f335a63f9b/volumes"
Jul 27 20:26:56.684616 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.684568    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"]
Jul 27 20:26:56.685258 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.685237    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09bba92c-0784-4133-a132-d1f335a63f9b" containerName="manager"
Jul 27 20:26:56.685258 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.685259    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="09bba92c-0784-4133-a132-d1f335a63f9b" containerName="manager"
Jul 27 20:26:56.685409 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.685389    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="09bba92c-0784-4133-a132-d1f335a63f9b" containerName="manager"
Jul 27 20:26:56.688716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.688685    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.692799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.692773    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m4xvk\""
Jul 27 20:26:56.711286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.711254    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"]
Jul 27 20:26:56.722426 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.722401    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.722539 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.722440    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cklch\" (UniqueName: \"kubernetes.io/projected/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-kube-api-access-cklch\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.823696 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.823660    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.823891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.823712    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cklch\" (UniqueName: \"kubernetes.io/projected/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-kube-api-access-cklch\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.824116 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.824095    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.853561 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.853529    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cklch\" (UniqueName: \"kubernetes.io/projected/b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c-kube-api-access-cklch\") pod \"kuadrant-operator-controller-manager-86878c769f-8wlh8\" (UID: \"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:56.998369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:56.998279    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:57.138579 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:57.138552    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"]
Jul 27 20:26:57.140383 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:26:57.140355    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cdf9c3_8736_4d3b_a5cb_d356d34e2e7c.slice/crio-0b4d754f7051d77164a5bb7056dc7d87e1dbd2ed48761f1c3b1493b69c542738 WatchSource:0}: Error finding container 0b4d754f7051d77164a5bb7056dc7d87e1dbd2ed48761f1c3b1493b69c542738: Status 404 returned error can't find the container with id 0b4d754f7051d77164a5bb7056dc7d87e1dbd2ed48761f1c3b1493b69c542738
Jul 27 20:26:57.529745 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:57.529706    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8" event={"ID":"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c","Type":"ContainerStarted","Data":"d12961259907179f6728aad57937e3157d01e2a7665b4f101fc6b0e6279b2d31"}
Jul 27 20:26:57.529936 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:57.529768    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8" event={"ID":"b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c","Type":"ContainerStarted","Data":"0b4d754f7051d77164a5bb7056dc7d87e1dbd2ed48761f1c3b1493b69c542738"}
Jul 27 20:26:57.529936 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:57.529810    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:26:57.551880 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:26:57.551804    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8" podStartSLOduration=1.551787939 podStartE2EDuration="1.551787939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:26:57.549840824 +0000 UTC m=+1383.644426290" watchObservedRunningTime="2026-07-27 20:26:57.551787939 +0000 UTC m=+1383.646373402"
Jul 27 20:27:08.537916 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:27:08.537880    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8wlh8"
Jul 27 20:28:54.419859 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:28:54.419830    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:28:54.424778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:28:54.424756    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:30:00.135632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.135591    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"]
Jul 27 20:30:00.139514 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.139488    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"
Jul 27 20:30:00.141836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.141807    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:30:00.146835 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.146807    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"]
Jul 27 20:30:00.153710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.153690    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"
Jul 27 20:30:00.505839 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.505797    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"]
Jul 27 20:30:00.507636 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:30:00.507607    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37340c9_9376_4a8f_8de9_8a33238d1ab7.slice/crio-5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d WatchSource:0}: Error finding container 5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d: Status 404 returned error can't find the container with id 5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d
Jul 27 20:30:00.509637 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.509622    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:30:00.520978 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:00.520954    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt" event={"ID":"c37340c9-9376-4a8f-8de9-8a33238d1ab7","Type":"ContainerStarted","Data":"5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d"}
Jul 27 20:30:04.546848 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:04.546810    2531 generic.go:356] "Generic (PLEG): container finished" podID="c37340c9-9376-4a8f-8de9-8a33238d1ab7" containerID="0039b2ea02fe1a436916f567db32a2a41d105e3c9c40bc3fee940a30fa9dfb05" exitCode=0
Jul 27 20:30:04.547302 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:04.546895    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt" event={"ID":"c37340c9-9376-4a8f-8de9-8a33238d1ab7","Type":"ContainerDied","Data":"0039b2ea02fe1a436916f567db32a2a41d105e3c9c40bc3fee940a30fa9dfb05"}
Jul 27 20:30:05.692731 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:05.692706    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"
Jul 27 20:30:06.559252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:06.559222    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt"
Jul 27 20:30:06.559252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:06.559229    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-2j6wt" event={"ID":"c37340c9-9376-4a8f-8de9-8a33238d1ab7","Type":"ContainerDied","Data":"5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d"}
Jul 27 20:30:06.559448 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:30:06.559275    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd8c498ca269efcc8e000f75a0797f7d941070c6c6a1e5d45d820a2c4b5d59d"
Jul 27 20:33:54.457995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:33:54.457965    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:33:54.462958 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:33:54.462931    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:34:46.357388 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.357344    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:34:46.358062 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.358047    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37340c9-9376-4a8f-8de9-8a33238d1ab7" containerName="cleanup"
Jul 27 20:34:46.358108 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.358065    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37340c9-9376-4a8f-8de9-8a33238d1ab7" containerName="cleanup"
Jul 27 20:34:46.358217 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.358204    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37340c9-9376-4a8f-8de9-8a33238d1ab7" containerName="cleanup"
Jul 27 20:34:46.361402 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.361384    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.363873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.363849    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9c50dc8e-openshift-default-dockercfg-qbw9d\""
Jul 27 20:34:46.379333 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.379304    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:34:46.380614 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380582    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380730 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380656    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpcf\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380790 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380764    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380809    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380872 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380851    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380918 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380896    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.380969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380937    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.381008 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.380977    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.381087 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.381068    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482080    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482126    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482190    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482210    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482250    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpcf\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482304    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482326    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482349    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482392    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482578    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482641    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.482846 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482829    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.483009 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482988    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.483083 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.482996    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.484701 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.484678    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.485081 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.485063    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.490538 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.490507    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.490630 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.490583    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpcf\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf\") pod \"e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") " pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.672831 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.672735    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:46.822201 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.822150    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:34:46.823793 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:34:46.823769    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7406c06d_ea81_4c3b_b190_351fa30e8ede.slice/crio-34d41dd872a5880570d0acb643cce3a7c126209c6a01f3b52139e1704bda0a48 WatchSource:0}: Error finding container 34d41dd872a5880570d0acb643cce3a7c126209c6a01f3b52139e1704bda0a48: Status 404 returned error can't find the container with id 34d41dd872a5880570d0acb643cce3a7c126209c6a01f3b52139e1704bda0a48
Jul 27 20:34:46.825978 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.825943    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:34:46.826078 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.826010    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:34:46.826078 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:46.826042    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:34:47.107808 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:47.107771    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerStarted","Data":"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"}
Jul 27 20:34:47.108010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:47.107822    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerStarted","Data":"34d41dd872a5880570d0acb643cce3a7c126209c6a01f3b52139e1704bda0a48"}
Jul 27 20:34:47.150163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:47.150106    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podStartSLOduration=1.150088076 podStartE2EDuration="1.150088076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:47.146146175 +0000 UTC m=+1853.240731655" watchObservedRunningTime="2026-07-27 20:34:47.150088076 +0000 UTC m=+1853.244673539"
Jul 27 20:34:47.673887 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:47.673850    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:47.678920 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:47.678889    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:48.115219 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:48.115169    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:48.116364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:48.116340    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:34:49.318385 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.318350    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:34:49.322600 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.322578    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.324952 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.324926    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9c50dc8e\""
Jul 27 20:34:49.325083 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.325028    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9c50dc8e-dockercfg-q945g\""
Jul 27 20:34:49.333431 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.333334    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:34:49.412404 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.412364    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bd7g\" (UniqueName: \"kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.412605 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.412462    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.513159 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.513121    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bd7g\" (UniqueName: \"kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.513372 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.513224    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.514075 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.514054    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.522889 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.522866    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bd7g\" (UniqueName: \"kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g\") pod \"payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") " pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.636551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.636445    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:49.784595 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:49.784552    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:34:49.785947 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:34:49.785917    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ab4832_2a1b_4469_91fe_1cefe3eb4c79.slice/crio-ae432dd2285a7dce69bcbe27fc08f33a8f480aa22bd518cda0157cb1d3333e4a WatchSource:0}: Error finding container ae432dd2285a7dce69bcbe27fc08f33a8f480aa22bd518cda0157cb1d3333e4a: Status 404 returned error can't find the container with id ae432dd2285a7dce69bcbe27fc08f33a8f480aa22bd518cda0157cb1d3333e4a
Jul 27 20:34:50.127565 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:50.127524    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" event={"ID":"64ab4832-2a1b-4469-91fe-1cefe3eb4c79","Type":"ContainerStarted","Data":"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"}
Jul 27 20:34:50.127760 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:50.127587    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" event={"ID":"64ab4832-2a1b-4469-91fe-1cefe3eb4c79","Type":"ContainerStarted","Data":"ae432dd2285a7dce69bcbe27fc08f33a8f480aa22bd518cda0157cb1d3333e4a"}
Jul 27 20:34:50.127865 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:50.127850    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:34:50.146218 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:34:50.146126    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" podStartSLOduration=1.146110798 podStartE2EDuration="1.146110798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:34:50.143514618 +0000 UTC m=+1856.238100081" watchObservedRunningTime="2026-07-27 20:34:50.146110798 +0000 UTC m=+1856.240696294"
Jul 27 20:35:00.198466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.198431    2531 generic.go:356] "Generic (PLEG): container finished" podID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerID="33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395" exitCode=0
Jul 27 20:35:00.198882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.198505    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerDied","Data":"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"}
Jul 27 20:35:00.198955 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.198927    2531 scope.go:117] "RemoveContainer" containerID="33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"
Jul 27 20:35:00.199985 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.199956    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:35:00.200069 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.200025    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:35:00.200105 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:00.200071    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:35:01.143390 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:01.143349    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:35:01.206116 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:01.206078    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerStarted","Data":"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"}
Jul 27 20:35:01.673521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:01.673483    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:01.674842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:01.674815    2531 patch_prober.go:28] interesting pod/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 20:35:01.674947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:01.674866    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 20:35:02.673406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:02.673359    2531 patch_prober.go:28] interesting pod/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 20:35:02.673807 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:02.673440    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 20:35:03.115554 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:03.115516    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:03.674187 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:03.674143    2531 patch_prober.go:28] interesting pod/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 20:35:03.674575 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:03.674250    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 20:35:04.673232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:04.673191    2531 patch_prober.go:28] interesting pod/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 20:35:04.673459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:04.673286    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 20:35:05.676878 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:05.676846    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:05.677819 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:05.677797    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:24.555934 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.555777    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:35:24.556479 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.556117    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" podUID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" containerName="payload-processing" containerID="cri-o://407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e" gracePeriod=30
Jul 27 20:35:24.825767 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.825741    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:35:24.882891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.882859    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bd7g\" (UniqueName: \"kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g\") pod \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") "
Jul 27 20:35:24.883073 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.882900    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume\") pod \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\" (UID: \"64ab4832-2a1b-4469-91fe-1cefe3eb4c79\") "
Jul 27 20:35:24.883307 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.883280    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64ab4832-2a1b-4469-91fe-1cefe3eb4c79" (UID: "64ab4832-2a1b-4469-91fe-1cefe3eb4c79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:35:24.884908 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.884885    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g" (OuterVolumeSpecName: "kube-api-access-6bd7g") pod "64ab4832-2a1b-4469-91fe-1cefe3eb4c79" (UID: "64ab4832-2a1b-4469-91fe-1cefe3eb4c79"). InnerVolumeSpecName "kube-api-access-6bd7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:35:24.984322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.984292    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bd7g\" (UniqueName: \"kubernetes.io/projected/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-kube-api-access-6bd7g\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:24.984322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:24.984318    2531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ab4832-2a1b-4469-91fe-1cefe3eb4c79-plugins-config-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:25.347268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.347235    2531 generic.go:356] "Generic (PLEG): container finished" podID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" containerID="407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e" exitCode=0
Jul 27 20:35:25.347268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.347268    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"
Jul 27 20:35:25.347494 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.347318    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" event={"ID":"64ab4832-2a1b-4469-91fe-1cefe3eb4c79","Type":"ContainerDied","Data":"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"}
Jul 27 20:35:25.347494 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.347359    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v" event={"ID":"64ab4832-2a1b-4469-91fe-1cefe3eb4c79","Type":"ContainerDied","Data":"ae432dd2285a7dce69bcbe27fc08f33a8f480aa22bd518cda0157cb1d3333e4a"}
Jul 27 20:35:25.347494 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.347390    2531 scope.go:117] "RemoveContainer" containerID="407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"
Jul 27 20:35:25.357903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.357887    2531 scope.go:117] "RemoveContainer" containerID="407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"
Jul 27 20:35:25.358170 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:35:25.358152    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e\": container with ID starting with 407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e not found: ID does not exist" containerID="407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"
Jul 27 20:35:25.358245 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.358190    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e"} err="failed to get container status \"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e\": rpc error: code = NotFound desc = could not find container \"407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e\": container with ID starting with 407be3d8a39c4099a09dec5578c42eeba5c0857e399d88d18e9a0b18bf83ba6e not found: ID does not exist"
Jul 27 20:35:25.369084 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.369060    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:35:25.371364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:25.371340    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c50dc8e-5fd9ffbc54-tgn9v"]
Jul 27 20:35:26.347960 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:26.347907    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" path="/var/lib/kubelet/pods/64ab4832-2a1b-4469-91fe-1cefe3eb4c79/volumes"
Jul 27 20:35:29.357381 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:29.357340    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:35:29.357970 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:29.357588    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy" containerID="cri-o://a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5" gracePeriod=30
Jul 27 20:35:34.622742 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.622683    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:34.684870 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.684841    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.684885    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.684927    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.684978    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.684995    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685056    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685112    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbpcf\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685129    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685144    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert\") pod \"7406c06d-ea81-4c3b-b190-351fa30e8ede\" (UID: \"7406c06d-ea81-4c3b-b190-351fa30e8ede\") "
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685151    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:35:34.685308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685284    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:35:34.685620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685444    2531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.685620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685464    2531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-workload-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.685620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685533    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:35:34.685620 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685613    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:35:34.686011 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.685965    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data" (OuterVolumeSpecName: "istio-data") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:35:34.687707 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.687663    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:35:34.687857 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.687826    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token" (OuterVolumeSpecName: "istio-token") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:35:34.687937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.687871    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf" (OuterVolumeSpecName: "kube-api-access-kbpcf") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "kube-api-access-kbpcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:35:34.687937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.687871    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7406c06d-ea81-4c3b-b190-351fa30e8ede" (UID: "7406c06d-ea81-4c3b-b190-351fa30e8ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:35:34.786789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786752    2531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-data\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.786789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786780    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpcf\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-kube-api-access-kbpcf\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.786789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786794    2531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-podinfo\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.787056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786806    2531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7406c06d-ea81-4c3b-b190-351fa30e8ede-istiod-ca-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.787056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786814    2531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-envoy\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.787056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786821    2531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7406c06d-ea81-4c3b-b190-351fa30e8ede-istio-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:34.787056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:34.786829    2531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7406c06d-ea81-4c3b-b190-351fa30e8ede-credential-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:35:35.406743 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.406708    2531 generic.go:356] "Generic (PLEG): container finished" podID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerID="a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5" exitCode=0
Jul 27 20:35:35.406924 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.406750    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerDied","Data":"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"}
Jul 27 20:35:35.406924 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.406782    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"
Jul 27 20:35:35.406924 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.406813    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt" event={"ID":"7406c06d-ea81-4c3b-b190-351fa30e8ede","Type":"ContainerDied","Data":"34d41dd872a5880570d0acb643cce3a7c126209c6a01f3b52139e1704bda0a48"}
Jul 27 20:35:35.406924 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.406837    2531 scope.go:117] "RemoveContainer" containerID="a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"
Jul 27 20:35:35.421486 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.421460    2531 scope.go:117] "RemoveContainer" containerID="33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"
Jul 27 20:35:35.430318 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.430301    2531 scope.go:117] "RemoveContainer" containerID="a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"
Jul 27 20:35:35.430554 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:35:35.430532    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5\": container with ID starting with a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5 not found: ID does not exist" containerID="a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"
Jul 27 20:35:35.430618 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.430560    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5"} err="failed to get container status \"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5\": rpc error: code = NotFound desc = could not find container \"a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5\": container with ID starting with a557e7080163be776a1a07ae2421ccdf60f282599d489df1d98f008e92c61fb5 not found: ID does not exist"
Jul 27 20:35:35.430618 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.430582    2531 scope.go:117] "RemoveContainer" containerID="33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"
Jul 27 20:35:35.430829 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:35:35.430810    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395\": container with ID starting with 33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395 not found: ID does not exist" containerID="33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"
Jul 27 20:35:35.430886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.430836    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395"} err="failed to get container status \"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395\": rpc error: code = NotFound desc = could not find container \"33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395\": container with ID starting with 33bbcca9ce4fe887c69f51f114e246dbf3ace9c48856634dff5499b6aa073395 not found: ID does not exist"
Jul 27 20:35:35.431285 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.431253    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:35:35.434954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:35.434934    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9c50dc8e-openshift-default-6ffb48587d-cm8gt"]
Jul 27 20:35:36.341493 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:35:36.341455    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" path="/var/lib/kubelet/pods/7406c06d-ea81-4c3b-b190-351fa30e8ede/volumes"
Jul 27 20:36:00.730808 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.730773    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"]
Jul 27 20:36:00.731272 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731257    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" containerName="payload-processing"
Jul 27 20:36:00.731341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731273    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" containerName="payload-processing"
Jul 27 20:36:00.731341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731290    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:00.731341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731297    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:00.731435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731383    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ab4832-2a1b-4469-91fe-1cefe3eb4c79" containerName="payload-processing"
Jul 27 20:36:00.731435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.731394    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:00.734419 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.734400    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:00.736706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.736683    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:36:00.742909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.742884    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"]
Jul 27 20:36:00.851432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.851401    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd\" (UID: \"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:00.952431 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.952394    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd\" (UID: \"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:00.953011 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:00.952986    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd\" (UID: \"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:01.047577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:01.047546    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:01.191818 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:01.191792    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"]
Jul 27 20:36:01.194063 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:36:01.194041    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cca86cd_4b62_4d68_ad0e_f353b4dff0a8.slice/crio-8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156 WatchSource:0}: Error finding container 8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156: Status 404 returned error can't find the container with id 8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156
Jul 27 20:36:01.554466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:01.554431    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd" event={"ID":"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8","Type":"ContainerStarted","Data":"8fe426a537d507ecabc127be6f13ef274485f2771b609c4165d9902e2d7bbd83"}
Jul 27 20:36:01.554634 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:01.554490    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd" event={"ID":"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8","Type":"ContainerStarted","Data":"8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156"}
Jul 27 20:36:01.571042 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:01.570989    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd" podStartSLOduration=1.570974788 podStartE2EDuration="1.570974788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:01.569379896 +0000 UTC m=+1927.663965363" watchObservedRunningTime="2026-07-27 20:36:01.570974788 +0000 UTC m=+1927.665560252"
Jul 27 20:36:02.562716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:02.562674    2531 generic.go:356] "Generic (PLEG): container finished" podID="5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" containerID="8fe426a537d507ecabc127be6f13ef274485f2771b609c4165d9902e2d7bbd83" exitCode=0
Jul 27 20:36:02.563093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:02.562760    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd" event={"ID":"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8","Type":"ContainerDied","Data":"8fe426a537d507ecabc127be6f13ef274485f2771b609c4165d9902e2d7bbd83"}
Jul 27 20:36:03.709194 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:03.709153    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:03.778740 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:03.778709    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca\") pod \"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8\" (UID: \"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8\") "
Jul 27 20:36:03.779058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:03.779036    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" (UID: "5cca86cd-4b62-4d68-ad0e-f353b4dff0a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:03.879993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:03.879903    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:36:04.575851 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:04.575820    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"
Jul 27 20:36:04.576033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:04.575843    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd" event={"ID":"5cca86cd-4b62-4d68-ad0e-f353b4dff0a8","Type":"ContainerDied","Data":"8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156"}
Jul 27 20:36:04.576033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:04.575926    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a687b87e1f0ab8f7b5fc622bca153fe95fd47852bc553bbba6259b8bdfcf156"
Jul 27 20:36:15.800785 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:15.800749    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"]
Jul 27 20:36:15.802793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:15.802771    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73a591cd-z2mtd"]
Jul 27 20:36:16.339860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:16.339825    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" path="/var/lib/kubelet/pods/5cca86cd-4b62-4d68-ad0e-f353b4dff0a8/volumes"
Jul 27 20:36:19.387643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.387598    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:19.388321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388297    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:19.388321 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388320    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:19.388506 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388358    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" containerName="revoke-keys"
Jul 27 20:36:19.388506 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388368    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" containerName="revoke-keys"
Jul 27 20:36:19.388506 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388499    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="7406c06d-ea81-4c3b-b190-351fa30e8ede" containerName="istio-proxy"
Jul 27 20:36:19.388661 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.388516    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cca86cd-4b62-4d68-ad0e-f353b4dff0a8" containerName="revoke-keys"
Jul 27 20:36:19.392059 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.392037    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.394578 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.394540    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-35ff59c0-serving-cert\""
Jul 27 20:36:19.401687 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.401661    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:19.530968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.530933    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dbw\" (UniqueName: \"kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.531146 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.531006    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.531146 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.531053    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.632341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.632310    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.632521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.632360    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.632521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.632407    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dbw\" (UniqueName: \"kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.633123 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.633100    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.635072 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.635036    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.641927 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.641857    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dbw\" (UniqueName: \"kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw\") pod \"maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.706621 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.706591    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:19.849401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.849365    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:19.851528 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:36:19.851497    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf7c460_0eda_484c_8b72_8ab4f782ff4f.slice/crio-57da10283b4199e0baac065f0a8235900ec6d6d65467bc683b4b220953636c18 WatchSource:0}: Error finding container 57da10283b4199e0baac065f0a8235900ec6d6d65467bc683b4b220953636c18: Status 404 returned error can't find the container with id 57da10283b4199e0baac065f0a8235900ec6d6d65467bc683b4b220953636c18
Jul 27 20:36:19.853360 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:19.853339    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:36:20.669717 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:20.669685    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" event={"ID":"ebf7c460-0eda-484c-8b72-8ab4f782ff4f","Type":"ContainerStarted","Data":"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"}
Jul 27 20:36:20.670103 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:20.669737    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" event={"ID":"ebf7c460-0eda-484c-8b72-8ab4f782ff4f","Type":"ContainerStarted","Data":"57da10283b4199e0baac065f0a8235900ec6d6d65467bc683b4b220953636c18"}
Jul 27 20:36:20.670103 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:20.669775    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:20.687629 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:20.687581    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" podStartSLOduration=1.088892811 podStartE2EDuration="1.687566162s" totalImagesPullingTime="598.673351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:20.686023155 +0000 UTC m=+1946.780608618" watchObservedRunningTime="2026-07-27 20:36:20.687566162 +0000 UTC m=+1946.782151627"
Jul 27 20:36:26.680130 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:26.680099    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:29.231844 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.231803    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"]
Jul 27 20:36:29.235688 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.235663    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:29.237973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.237948    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:36:29.243112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.243087    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"]
Jul 27 20:36:29.333198 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.333137    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9\" (UID: \"59976ce3-cf03-48cb-9da6-f293fe830f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:29.433963 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.433919    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9\" (UID: \"59976ce3-cf03-48cb-9da6-f293fe830f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:29.434598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.434572    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9\" (UID: \"59976ce3-cf03-48cb-9da6-f293fe830f6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:29.551547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.551503    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:29.691232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.691202    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"]
Jul 27 20:36:29.692918 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:36:29.692892    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59976ce3_cf03_48cb_9da6_f293fe830f6c.slice/crio-f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81 WatchSource:0}: Error finding container f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81: Status 404 returned error can't find the container with id f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81
Jul 27 20:36:29.725098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:29.725058    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9" event={"ID":"59976ce3-cf03-48cb-9da6-f293fe830f6c","Type":"ContainerStarted","Data":"f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81"}
Jul 27 20:36:30.732291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:30.732204    2531 generic.go:356] "Generic (PLEG): container finished" podID="59976ce3-cf03-48cb-9da6-f293fe830f6c" containerID="31ceae5c203bd732cf7bcf95099e687ee21882ed67f0e66fa631c6eeaada4765" exitCode=0
Jul 27 20:36:30.732291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:30.732266    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9" event={"ID":"59976ce3-cf03-48cb-9da6-f293fe830f6c","Type":"ContainerDied","Data":"31ceae5c203bd732cf7bcf95099e687ee21882ed67f0e66fa631c6eeaada4765"}
Jul 27 20:36:31.883726 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:31.883694    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:31.956856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:31.956817    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca\") pod \"59976ce3-cf03-48cb-9da6-f293fe830f6c\" (UID: \"59976ce3-cf03-48cb-9da6-f293fe830f6c\") "
Jul 27 20:36:31.957287 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:31.957253    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59976ce3-cf03-48cb-9da6-f293fe830f6c" (UID: "59976ce3-cf03-48cb-9da6-f293fe830f6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:32.058363 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:32.058327    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59976ce3-cf03-48cb-9da6-f293fe830f6c-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:36:32.748993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:32.748899    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9" event={"ID":"59976ce3-cf03-48cb-9da6-f293fe830f6c","Type":"ContainerDied","Data":"f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81"}
Jul 27 20:36:32.748993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:32.748931    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:32.748993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:32.748953    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9dbc7a993fd0f1babbaaa154f9b8763945c4f52fcd3cdeaccafa6cd09f03a81"
Jul 27 20:36:40.507110 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.507042    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:40.509488 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.509453    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" podUID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" containerName="maas-api" containerID="cri-o://a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b" gracePeriod=30
Jul 27 20:36:40.781268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.781190    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:40.802239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.801852    2531 generic.go:356] "Generic (PLEG): container finished" podID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" containerID="a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b" exitCode=0
Jul 27 20:36:40.802239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.801915    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"
Jul 27 20:36:40.802239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.801972    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" event={"ID":"ebf7c460-0eda-484c-8b72-8ab4f782ff4f","Type":"ContainerDied","Data":"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"}
Jul 27 20:36:40.802239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.802018    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf" event={"ID":"ebf7c460-0eda-484c-8b72-8ab4f782ff4f","Type":"ContainerDied","Data":"57da10283b4199e0baac065f0a8235900ec6d6d65467bc683b4b220953636c18"}
Jul 27 20:36:40.802239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.802046    2531 scope.go:117] "RemoveContainer" containerID="a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"
Jul 27 20:36:40.812257 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.812232    2531 scope.go:117] "RemoveContainer" containerID="a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"
Jul 27 20:36:40.812560 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:36:40.812540    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b\": container with ID starting with a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b not found: ID does not exist" containerID="a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"
Jul 27 20:36:40.812619 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.812572    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b"} err="failed to get container status \"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b\": rpc error: code = NotFound desc = could not find container \"a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b\": container with ID starting with a8381fc24add10736e8654438b8df151fba389b5230f27a91c6fdd09a16b388b not found: ID does not exist"
Jul 27 20:36:40.845741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.845693    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6dbw\" (UniqueName: \"kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw\") pod \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") "
Jul 27 20:36:40.845741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.845743    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca\") pod \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") "
Jul 27 20:36:40.846012 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.845832    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls\") pod \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\" (UID: \"ebf7c460-0eda-484c-8b72-8ab4f782ff4f\") "
Jul 27 20:36:40.846492 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.846449    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebf7c460-0eda-484c-8b72-8ab4f782ff4f" (UID: "ebf7c460-0eda-484c-8b72-8ab4f782ff4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:40.848463 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.848418    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebf7c460-0eda-484c-8b72-8ab4f782ff4f" (UID: "ebf7c460-0eda-484c-8b72-8ab4f782ff4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:40.848596 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.848456    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw" (OuterVolumeSpecName: "kube-api-access-k6dbw") pod "ebf7c460-0eda-484c-8b72-8ab4f782ff4f" (UID: "ebf7c460-0eda-484c-8b72-8ab4f782ff4f"). InnerVolumeSpecName "kube-api-access-k6dbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:40.947160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.947128    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dbw\" (UniqueName: \"kubernetes.io/projected/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-kube-api-access-k6dbw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:36:40.947160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.947158    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:36:40.947160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:40.947199    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebf7c460-0eda-484c-8b72-8ab4f782ff4f-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:36:41.133276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:41.133239    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:41.135575 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:41.135539    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35ff59c0-f79ddb58-7hstf"]
Jul 27 20:36:42.339576 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:42.339547    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" path="/var/lib/kubelet/pods/ebf7c460-0eda-484c-8b72-8ab4f782ff4f/volumes"
Jul 27 20:36:44.330835 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:44.330796    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"]
Jul 27 20:36:44.337104 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:36:44.337073    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="59976ce3-cf03-48cb-9da6-f293fe830f6c" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"
Jul 27 20:36:44.339709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:44.339681    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff59c0-82ls9"]
Jul 27 20:36:46.340886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:46.340812    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59976ce3-cf03-48cb-9da6-f293fe830f6c" path="/var/lib/kubelet/pods/59976ce3-cf03-48cb-9da6-f293fe830f6c/volumes"
Jul 27 20:36:49.104532 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.104493    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:36:49.105858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.105828    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" containerName="maas-api"
Jul 27 20:36:49.105858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.105859    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" containerName="maas-api"
Jul 27 20:36:49.106047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.105879    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59976ce3-cf03-48cb-9da6-f293fe830f6c" containerName="revoke-keys"
Jul 27 20:36:49.106047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.105888    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="59976ce3-cf03-48cb-9da6-f293fe830f6c" containerName="revoke-keys"
Jul 27 20:36:49.106047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.106030    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="59976ce3-cf03-48cb-9da6-f293fe830f6c" containerName="revoke-keys"
Jul 27 20:36:49.106047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.106046    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf7c460-0eda-484c-8b72-8ab4f782ff4f" containerName="maas-api"
Jul 27 20:36:49.111026 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.111006    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.113557 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.113537    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0d589065-serving-cert\""
Jul 27 20:36:49.121684 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.120971    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:36:49.231902 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.231871    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.232093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.231917    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566q4\" (UniqueName: \"kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.232093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.231941    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.333452 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.333418    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.333627 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.333479    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566q4\" (UniqueName: \"kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.333627 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.333511    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.333723 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:36:49.333659    2531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-serving-cert: secret "maas-api-e2e-derive-0d589065-serving-cert" not found
Jul 27 20:36:49.333723 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:36:49.333722    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls podName:43180e9a-cc8e-4333-afa5-794a7b7f873f nodeName:}" failed. No retries permitted until 2026-07-27 20:36:49.833708648 +0000 UTC m=+1975.928294095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls") pod "maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" (UID: "43180e9a-cc8e-4333-afa5-794a7b7f873f") : secret "maas-api-e2e-derive-0d589065-serving-cert" not found
Jul 27 20:36:49.334099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.334078    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.342319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.342282    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-566q4\" (UniqueName: \"kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.838734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.838696    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:49.841600 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:49.841567    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") pod \"maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:50.025205 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:50.025157    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:50.172652 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:50.172619    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:36:50.173657 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:36:50.173622    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43180e9a_cc8e_4333_afa5_794a7b7f873f.slice/crio-80ac95a9ca3fbe4275f31530b3099cbeecf3049636518bfafa54bbb4a2dda0a6 WatchSource:0}: Error finding container 80ac95a9ca3fbe4275f31530b3099cbeecf3049636518bfafa54bbb4a2dda0a6: Status 404 returned error can't find the container with id 80ac95a9ca3fbe4275f31530b3099cbeecf3049636518bfafa54bbb4a2dda0a6
Jul 27 20:36:50.863753 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:50.863724    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" event={"ID":"43180e9a-cc8e-4333-afa5-794a7b7f873f","Type":"ContainerStarted","Data":"80ac95a9ca3fbe4275f31530b3099cbeecf3049636518bfafa54bbb4a2dda0a6"}
Jul 27 20:36:51.871363 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:51.871324    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" event={"ID":"43180e9a-cc8e-4333-afa5-794a7b7f873f","Type":"ContainerStarted","Data":"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"}
Jul 27 20:36:51.871805 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:51.871511    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:36:51.889312 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:51.889264    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" podStartSLOduration=2.252007575 podStartE2EDuration="2.88925027s" totalImagesPullingTime="637.242695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:51.886662685 +0000 UTC m=+1977.981248149" watchObservedRunningTime="2026-07-27 20:36:51.88925027 +0000 UTC m=+1977.983835733"
Jul 27 20:36:57.884041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:36:57.884009    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:37:13.822093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.822056    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"]
Jul 27 20:37:13.843459 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.843419    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"]
Jul 27 20:37:13.843643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.843546    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:13.848551 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.848494    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:37:13.863231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.863198    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d589065-plr79\" (UID: \"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:13.964363 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.964324    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d589065-plr79\" (UID: \"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:13.964992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:13.964971    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d589065-plr79\" (UID: \"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:14.165693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:14.165610    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:14.296762 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:14.296597    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"]
Jul 27 20:37:14.301330 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:37:14.301284    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cf523f_b5d3_44b7_bd7e_4d5e7df9f4c2.slice/crio-c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70 WatchSource:0}: Error finding container c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70: Status 404 returned error can't find the container with id c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70
Jul 27 20:37:14.998523 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:14.998484    2531 generic.go:356] "Generic (PLEG): container finished" podID="09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" containerID="e4b2ba74b19abfce80ec18f41aa486ad41c8c391177c33d10bd4012cfe4c5352" exitCode=0
Jul 27 20:37:14.998965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:14.998562    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79" event={"ID":"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2","Type":"ContainerDied","Data":"e4b2ba74b19abfce80ec18f41aa486ad41c8c391177c33d10bd4012cfe4c5352"}
Jul 27 20:37:14.998965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:14.998611    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79" event={"ID":"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2","Type":"ContainerStarted","Data":"c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70"}
Jul 27 20:37:16.140052 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:16.140027    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:16.188162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:16.188126    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca\") pod \"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2\" (UID: \"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2\") "
Jul 27 20:37:16.188495 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:16.188471    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" (UID: "09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:37:16.289632 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:16.289596    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:37:17.011076 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:17.011045    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"
Jul 27 20:37:17.011262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:17.011078    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79" event={"ID":"09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2","Type":"ContainerDied","Data":"c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70"}
Jul 27 20:37:17.011262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:17.011129    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c220778fb8068655570d13b0985353cf9073ed8a9908075c4b1757503b304a70"
Jul 27 20:37:24.213770 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.213719    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:37:24.214246 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.213977    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" podUID="43180e9a-cc8e-4333-afa5-794a7b7f873f" containerName="maas-api" containerID="cri-o://55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354" gracePeriod=30
Jul 27 20:37:24.476973 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.476906    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:37:24.574666 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.574633    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca\") pod \"43180e9a-cc8e-4333-afa5-794a7b7f873f\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") "
Jul 27 20:37:24.574860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.574723    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-566q4\" (UniqueName: \"kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4\") pod \"43180e9a-cc8e-4333-afa5-794a7b7f873f\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") "
Jul 27 20:37:24.574860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.574764    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") pod \"43180e9a-cc8e-4333-afa5-794a7b7f873f\" (UID: \"43180e9a-cc8e-4333-afa5-794a7b7f873f\") "
Jul 27 20:37:24.574980 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.574958    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43180e9a-cc8e-4333-afa5-794a7b7f873f" (UID: "43180e9a-cc8e-4333-afa5-794a7b7f873f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:37:24.575158 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.575141    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43180e9a-cc8e-4333-afa5-794a7b7f873f-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:37:24.576840 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.576818    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43180e9a-cc8e-4333-afa5-794a7b7f873f" (UID: "43180e9a-cc8e-4333-afa5-794a7b7f873f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:37:24.576938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.576846    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4" (OuterVolumeSpecName: "kube-api-access-566q4") pod "43180e9a-cc8e-4333-afa5-794a7b7f873f" (UID: "43180e9a-cc8e-4333-afa5-794a7b7f873f"). InnerVolumeSpecName "kube-api-access-566q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:37:24.676540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.676507    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566q4\" (UniqueName: \"kubernetes.io/projected/43180e9a-cc8e-4333-afa5-794a7b7f873f-kube-api-access-566q4\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:37:24.676540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:24.676537    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43180e9a-cc8e-4333-afa5-794a7b7f873f-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:37:25.056046 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.056012    2531 generic.go:356] "Generic (PLEG): container finished" podID="43180e9a-cc8e-4333-afa5-794a7b7f873f" containerID="55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354" exitCode=0
Jul 27 20:37:25.056271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.056060    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"
Jul 27 20:37:25.056271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.056104    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" event={"ID":"43180e9a-cc8e-4333-afa5-794a7b7f873f","Type":"ContainerDied","Data":"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"}
Jul 27 20:37:25.056271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.056156    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj" event={"ID":"43180e9a-cc8e-4333-afa5-794a7b7f873f","Type":"ContainerDied","Data":"80ac95a9ca3fbe4275f31530b3099cbeecf3049636518bfafa54bbb4a2dda0a6"}
Jul 27 20:37:25.056271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.056195    2531 scope.go:117] "RemoveContainer" containerID="55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"
Jul 27 20:37:25.065813 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.065791    2531 scope.go:117] "RemoveContainer" containerID="55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"
Jul 27 20:37:25.066111 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:37:25.066085    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354\": container with ID starting with 55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354 not found: ID does not exist" containerID="55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"
Jul 27 20:37:25.066255 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.066116    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354"} err="failed to get container status \"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354\": rpc error: code = NotFound desc = could not find container \"55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354\": container with ID starting with 55c1e29d921230f85d123ace2ce97e07488eb0a013c48eba0bb5a31d7a4d5354 not found: ID does not exist"
Jul 27 20:37:25.082210 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.082152    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:37:25.087550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:25.087516    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d589065-cd9f46d9c-lgbcj"]
Jul 27 20:37:26.341369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:26.341322    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43180e9a-cc8e-4333-afa5-794a7b7f873f" path="/var/lib/kubelet/pods/43180e9a-cc8e-4333-afa5-794a7b7f873f/volumes"
Jul 27 20:37:28.891048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:28.891015    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"]
Jul 27 20:37:28.893031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:28.893009    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d589065-plr79"]
Jul 27 20:37:30.339773 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:30.339734    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" path="/var/lib/kubelet/pods/09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2/volumes"
Jul 27 20:37:57.242369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:57.242334    2531 generic.go:356] "Generic (PLEG): container finished" podID="8520ebf0-6848-4f90-8ba2-a84ae16136a1" containerID="66796598df6ef5dfb280b24bd3d85e97569ce3aa3a1a2564fc2f04efc1c4b7d5" exitCode=1
Jul 27 20:37:57.242882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:57.242394    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld" event={"ID":"8520ebf0-6848-4f90-8ba2-a84ae16136a1","Type":"ContainerDied","Data":"66796598df6ef5dfb280b24bd3d85e97569ce3aa3a1a2564fc2f04efc1c4b7d5"}
Jul 27 20:37:57.242882 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:57.242812    2531 scope.go:117] "RemoveContainer" containerID="66796598df6ef5dfb280b24bd3d85e97569ce3aa3a1a2564fc2f04efc1c4b7d5"
Jul 27 20:37:58.250975 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:58.250937    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld" event={"ID":"8520ebf0-6848-4f90-8ba2-a84ae16136a1","Type":"ContainerStarted","Data":"4a3b3a6572bf9c979dde9cc522d9200ed515a254085bbe2ffb0b530477b6c271"}
Jul 27 20:37:58.251379 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:37:58.251138    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:38:29.261239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:38:29.261143    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pdjld"
Jul 27 20:38:54.500235 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:38:54.500196    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:38:54.506291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:38:54.506263    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:39:35.347428 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347394    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"]
Jul 27 20:39:35.347939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347883    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" containerName="revoke-keys"
Jul 27 20:39:35.347939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347897    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" containerName="revoke-keys"
Jul 27 20:39:35.347939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347906    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43180e9a-cc8e-4333-afa5-794a7b7f873f" containerName="maas-api"
Jul 27 20:39:35.347939 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347912    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="43180e9a-cc8e-4333-afa5-794a7b7f873f" containerName="maas-api"
Jul 27 20:39:35.348107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.347987    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="09cf523f-b5d3-44b7-bd7e-4d5e7df9f4c2" containerName="revoke-keys"
Jul 27 20:39:35.348107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.348000    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="43180e9a-cc8e-4333-afa5-794a7b7f873f" containerName="maas-api"
Jul 27 20:39:35.351436 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.351409    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:35.355383 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.355357    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:39:35.363081 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.363054    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"]
Jul 27 20:39:35.365589 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.365560    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf\" (UID: \"ae7305f6-e844-4777-862f-f587aeffd1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:35.466741 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.466707    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf\" (UID: \"ae7305f6-e844-4777-862f-f587aeffd1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:35.467436 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.467413    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf\" (UID: \"ae7305f6-e844-4777-862f-f587aeffd1ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:35.663038 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.662967    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:35.813079 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:35.813026    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"]
Jul 27 20:39:35.813664 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:39:35.813636    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7305f6_e844_4777_862f_f587aeffd1ef.slice/crio-22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0 WatchSource:0}: Error finding container 22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0: Status 404 returned error can't find the container with id 22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0
Jul 27 20:39:36.809224 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:36.809162    2531 generic.go:356] "Generic (PLEG): container finished" podID="ae7305f6-e844-4777-862f-f587aeffd1ef" containerID="8caf9a720f5850a5e4e8f55608369ae1f9e56f9f364b3c7f2e81df2722c549f0" exitCode=0
Jul 27 20:39:36.809630 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:36.809223    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf" event={"ID":"ae7305f6-e844-4777-862f-f587aeffd1ef","Type":"ContainerDied","Data":"8caf9a720f5850a5e4e8f55608369ae1f9e56f9f364b3c7f2e81df2722c549f0"}
Jul 27 20:39:36.809630 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:36.809286    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf" event={"ID":"ae7305f6-e844-4777-862f-f587aeffd1ef","Type":"ContainerStarted","Data":"22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0"}
Jul 27 20:39:37.955814 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:37.955785    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:37.992071 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:37.992038    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca\") pod \"ae7305f6-e844-4777-862f-f587aeffd1ef\" (UID: \"ae7305f6-e844-4777-862f-f587aeffd1ef\") "
Jul 27 20:39:37.992406 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:37.992384    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae7305f6-e844-4777-862f-f587aeffd1ef" (UID: "ae7305f6-e844-4777-862f-f587aeffd1ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:39:37.992521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:37.992509    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7305f6-e844-4777-862f-f587aeffd1ef-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:39:38.822333 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:38.822298    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"
Jul 27 20:39:38.822502 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:38.822333    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf" event={"ID":"ae7305f6-e844-4777-862f-f587aeffd1ef","Type":"ContainerDied","Data":"22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0"}
Jul 27 20:39:38.822502 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:38.822384    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22926eb71c0a47fc0f20c95b55b97fd1abcd40c21ed0c53dd95bc8a8758906b0"
Jul 27 20:39:50.474465 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:50.474387    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"]
Jul 27 20:39:50.479521 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:50.479493    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5b6e45fe-d6bqf"]
Jul 27 20:39:52.344043 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:39:52.343997    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7305f6-e844-4777-862f-f587aeffd1ef" path="/var/lib/kubelet/pods/ae7305f6-e844-4777-862f-f587aeffd1ef/volumes"
Jul 27 20:41:01.735899 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.735862    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"]
Jul 27 20:41:01.736462 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.736442    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7305f6-e844-4777-862f-f587aeffd1ef" containerName="revoke-keys"
Jul 27 20:41:01.736462 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.736458    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7305f6-e844-4777-862f-f587aeffd1ef" containerName="revoke-keys"
Jul 27 20:41:01.736597 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.736536    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7305f6-e844-4777-862f-f587aeffd1ef" containerName="revoke-keys"
Jul 27 20:41:01.739828 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.739797    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:01.742030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.742003    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:41:01.747763 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.747740    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"]
Jul 27 20:41:01.814862 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.814832    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-030d0aef-r45vl\" (UID: \"d4ec97aa-eb60-4e10-8f40-b7b1a749224a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:01.915719 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.915682    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-030d0aef-r45vl\" (UID: \"d4ec97aa-eb60-4e10-8f40-b7b1a749224a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:01.916339 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:01.916320    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-030d0aef-r45vl\" (UID: \"d4ec97aa-eb60-4e10-8f40-b7b1a749224a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:02.054339 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:02.054304    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:02.198289 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:02.198261    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"]
Jul 27 20:41:02.200382 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:41:02.200353    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ec97aa_eb60_4e10_8f40_b7b1a749224a.slice/crio-7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4 WatchSource:0}: Error finding container 7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4: Status 404 returned error can't find the container with id 7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4
Jul 27 20:41:02.322053 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:02.321973    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl" event={"ID":"d4ec97aa-eb60-4e10-8f40-b7b1a749224a","Type":"ContainerStarted","Data":"7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4"}
Jul 27 20:41:03.328646 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:03.328597    2531 generic.go:356] "Generic (PLEG): container finished" podID="d4ec97aa-eb60-4e10-8f40-b7b1a749224a" containerID="f132d140ca484a00dcea6a6398c955b1019c86f702a7919a60fa391363dce26f" exitCode=0
Jul 27 20:41:03.329038 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:03.328658    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl" event={"ID":"d4ec97aa-eb60-4e10-8f40-b7b1a749224a","Type":"ContainerDied","Data":"f132d140ca484a00dcea6a6398c955b1019c86f702a7919a60fa391363dce26f"}
Jul 27 20:41:04.476122 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:04.476097    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:04.542927 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:04.542895    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca\") pod \"d4ec97aa-eb60-4e10-8f40-b7b1a749224a\" (UID: \"d4ec97aa-eb60-4e10-8f40-b7b1a749224a\") "
Jul 27 20:41:04.543251 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:04.543228    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4ec97aa-eb60-4e10-8f40-b7b1a749224a" (UID: "d4ec97aa-eb60-4e10-8f40-b7b1a749224a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:41:04.643935 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:04.643859    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ec97aa-eb60-4e10-8f40-b7b1a749224a-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:41:05.344738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:05.344703    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"
Jul 27 20:41:05.344887 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:05.344711    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl" event={"ID":"d4ec97aa-eb60-4e10-8f40-b7b1a749224a","Type":"ContainerDied","Data":"7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4"}
Jul 27 20:41:05.344887 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:05.344860    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a82429bf2c42d5872ca049d67155e2086f7cf200fb88379238189151aff7ef4"
Jul 27 20:41:21.806601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:21.806475    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"]
Jul 27 20:41:21.808804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:21.808775    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-030d0aef-r45vl"]
Jul 27 20:41:22.340186 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:41:22.340150    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ec97aa-eb60-4e10-8f40-b7b1a749224a" path="/var/lib/kubelet/pods/d4ec97aa-eb60-4e10-8f40-b7b1a749224a/volumes"
Jul 27 20:42:54.620107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:42:54.620032    2531 scope.go:117] "RemoveContainer" containerID="31ceae5c203bd732cf7bcf95099e687ee21882ed67f0e66fa631c6eeaada4765"
Jul 27 20:42:54.628770 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:42:54.628746    2531 scope.go:117] "RemoveContainer" containerID="8fe426a537d507ecabc127be6f13ef274485f2771b609c4165d9902e2d7bbd83"
Jul 27 20:43:19.206285 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.206252    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:43:19.206921 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.206899    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec97aa-eb60-4e10-8f40-b7b1a749224a" containerName="revoke-keys"
Jul 27 20:43:19.207030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.206923    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec97aa-eb60-4e10-8f40-b7b1a749224a" containerName="revoke-keys"
Jul 27 20:43:19.207086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.207062    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ec97aa-eb60-4e10-8f40-b7b1a749224a" containerName="revoke-keys"
Jul 27 20:43:19.210645 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.210620    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.213541 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.213516    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a12508-serving-cert\""
Jul 27 20:43:19.222264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.222201    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:43:19.367231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.367200    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnl2\" (UniqueName: \"kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.367393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.367257    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.367393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.367298    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.468555 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.468475    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.468733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.468699    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnl2\" (UniqueName: \"kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.468804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.468755    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.469211 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.469158    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.471098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.471080    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.481100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.481079    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnl2\" (UniqueName: \"kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2\") pod \"maas-api-e2e-inf-a-a12508-65478c45-bg2vn\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.526602 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.526575    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:19.678386 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.678360    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:43:19.680432 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:43:19.680401    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700c3c8a_8660_43c6_8fe9_d54e31ac6956.slice/crio-2461b5886ccc462f31cefb1024efb6bd2f2e8ccc3b55f6f7221f3d83b0a37baa WatchSource:0}: Error finding container 2461b5886ccc462f31cefb1024efb6bd2f2e8ccc3b55f6f7221f3d83b0a37baa: Status 404 returned error can't find the container with id 2461b5886ccc462f31cefb1024efb6bd2f2e8ccc3b55f6f7221f3d83b0a37baa
Jul 27 20:43:19.681933 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:19.681913    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:43:20.117786 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:20.117754    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" event={"ID":"700c3c8a-8660-43c6-8fe9-d54e31ac6956","Type":"ContainerStarted","Data":"2461b5886ccc462f31cefb1024efb6bd2f2e8ccc3b55f6f7221f3d83b0a37baa"}
Jul 27 20:43:21.125123 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:21.125085    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" event={"ID":"700c3c8a-8660-43c6-8fe9-d54e31ac6956","Type":"ContainerStarted","Data":"5515c17aa40be2ca60289b1c5e6b93a2a2a246f13a89695739986307a75eb48c"}
Jul 27 20:43:21.125540 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:21.125162    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:21.162880 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:21.162824    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" podStartSLOduration=1.721648957 podStartE2EDuration="2.162808817s" totalImagesPullingTime="441.15986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:21.160143411 +0000 UTC m=+2367.254728876" watchObservedRunningTime="2026-07-27 20:43:21.162808817 +0000 UTC m=+2367.257394281"
Jul 27 20:43:27.137460 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.137427    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:43:27.203954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.203925    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:43:27.208273 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.208249    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.210996 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.210975    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eb7a86-serving-cert\""
Jul 27 20:43:27.222118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.221917    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:43:27.349005 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.348962    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.349215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.349022    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.349215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.349043    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlscz\" (UniqueName: \"kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.450432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.450355    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.450432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.450399    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.450432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.450427    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlscz\" (UniqueName: \"kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.450698 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:43:27.450531    2531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-serving-cert: secret "maas-api-e2e-inf-b-eb7a86-serving-cert" not found
Jul 27 20:43:27.450698 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:43:27.450662    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls podName:483864b3-631c-455c-8b0d-53a8f60ebf3b nodeName:}" failed. No retries permitted until 2026-07-27 20:43:27.950639862 +0000 UTC m=+2374.045225320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls") pod "maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" (UID: "483864b3-631c-455c-8b0d-53a8f60ebf3b") : secret "maas-api-e2e-inf-b-eb7a86-serving-cert" not found
Jul 27 20:43:27.451044 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.451024    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.461270 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.461243    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlscz\" (UniqueName: \"kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.956513 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.956476    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:27.959213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:27.959164    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:28.122281 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:28.122245    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:28.271309 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:28.271279    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:43:28.274685 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:43:28.274655    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483864b3_631c_455c_8b0d_53a8f60ebf3b.slice/crio-468c42ea36e48c8d263047e471cdc1ffd76c4c764bb71c5ff8bbbdaed2112c9c WatchSource:0}: Error finding container 468c42ea36e48c8d263047e471cdc1ffd76c4c764bb71c5ff8bbbdaed2112c9c: Status 404 returned error can't find the container with id 468c42ea36e48c8d263047e471cdc1ffd76c4c764bb71c5ff8bbbdaed2112c9c
Jul 27 20:43:29.172372 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:29.172326    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" event={"ID":"483864b3-631c-455c-8b0d-53a8f60ebf3b","Type":"ContainerStarted","Data":"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"}
Jul 27 20:43:29.172533 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:29.172393    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:29.172533 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:29.172415    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" event={"ID":"483864b3-631c-455c-8b0d-53a8f60ebf3b","Type":"ContainerStarted","Data":"468c42ea36e48c8d263047e471cdc1ffd76c4c764bb71c5ff8bbbdaed2112c9c"}
Jul 27 20:43:29.191322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:29.191266    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" podStartSLOduration=1.914573793 podStartE2EDuration="2.191247339s" totalImagesPullingTime="276.673546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:43:29.190287578 +0000 UTC m=+2375.284873042" watchObservedRunningTime="2026-07-27 20:43:29.191247339 +0000 UTC m=+2375.285832805"
Jul 27 20:43:35.183535 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:35.183501    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:43:54.538082 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:54.538054    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:43:54.547607 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:54.547585    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:43:54.657489 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:43:54.657457    2531 scope.go:117] "RemoveContainer" containerID="e4b2ba74b19abfce80ec18f41aa486ad41c8c391177c33d10bd4012cfe4c5352"
Jul 27 20:44:37.030937 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.030895    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"]
Jul 27 20:44:37.034852 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.034825    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:37.037249 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.037228    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:44:37.042919 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.042893    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"]
Jul 27 20:44:37.125054 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.125015    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw\" (UID: \"7da04c2c-c777-4221-b857-81c7387af737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:37.225717 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.225685    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw\" (UID: \"7da04c2c-c777-4221-b857-81c7387af737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:37.226335 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.226315    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw\" (UID: \"7da04c2c-c777-4221-b857-81c7387af737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:37.350404 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.350332    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:37.490048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.490011    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"]
Jul 27 20:44:37.491618 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:44:37.491577    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da04c2c_c777_4221_b857_81c7387af737.slice/crio-3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57 WatchSource:0}: Error finding container 3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57: Status 404 returned error can't find the container with id 3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57
Jul 27 20:44:37.578418 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:37.578382    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw" event={"ID":"7da04c2c-c777-4221-b857-81c7387af737","Type":"ContainerStarted","Data":"3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57"}
Jul 27 20:44:38.586035 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:38.585993    2531 generic.go:356] "Generic (PLEG): container finished" podID="7da04c2c-c777-4221-b857-81c7387af737" containerID="524ec8d61ea77b5bccbbe37729faa963213e4214789b5a87be2337fb9224348a" exitCode=0
Jul 27 20:44:38.586432 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:38.586071    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw" event={"ID":"7da04c2c-c777-4221-b857-81c7387af737","Type":"ContainerDied","Data":"524ec8d61ea77b5bccbbe37729faa963213e4214789b5a87be2337fb9224348a"}
Jul 27 20:44:39.735512 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:39.735485    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:39.852941 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:39.852865    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca\") pod \"7da04c2c-c777-4221-b857-81c7387af737\" (UID: \"7da04c2c-c777-4221-b857-81c7387af737\") "
Jul 27 20:44:39.853231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:39.853210    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7da04c2c-c777-4221-b857-81c7387af737" (UID: "7da04c2c-c777-4221-b857-81c7387af737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:44:39.954456 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:39.954421    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da04c2c-c777-4221-b857-81c7387af737-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:44:40.598399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:40.598292    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"
Jul 27 20:44:40.598399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:40.598327    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw" event={"ID":"7da04c2c-c777-4221-b857-81c7387af737","Type":"ContainerDied","Data":"3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57"}
Jul 27 20:44:40.598399 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:40.598396    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3726c6d030cac234363602be0d02a3e77bd34a90e933276bf5c218bd7a266c57"
Jul 27 20:44:47.500807 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.500772    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:44:47.501286 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.501060    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" podUID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" containerName="maas-api" containerID="cri-o://5515c17aa40be2ca60289b1c5e6b93a2a2a246f13a89695739986307a75eb48c" gracePeriod=30
Jul 27 20:44:47.640664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.640400    2531 generic.go:356] "Generic (PLEG): container finished" podID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" containerID="5515c17aa40be2ca60289b1c5e6b93a2a2a246f13a89695739986307a75eb48c" exitCode=0
Jul 27 20:44:47.640664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.640529    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" event={"ID":"700c3c8a-8660-43c6-8fe9-d54e31ac6956","Type":"ContainerDied","Data":"5515c17aa40be2ca60289b1c5e6b93a2a2a246f13a89695739986307a75eb48c"}
Jul 27 20:44:47.768799 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.768773    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:44:47.933129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.933097    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca\") pod \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") "
Jul 27 20:44:47.933129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.933135    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls\") pod \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") "
Jul 27 20:44:47.933349 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.933258    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnnl2\" (UniqueName: \"kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2\") pod \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\" (UID: \"700c3c8a-8660-43c6-8fe9-d54e31ac6956\") "
Jul 27 20:44:47.933560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.933529    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "700c3c8a-8660-43c6-8fe9-d54e31ac6956" (UID: "700c3c8a-8660-43c6-8fe9-d54e31ac6956"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:44:47.935705 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.935676    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2" (OuterVolumeSpecName: "kube-api-access-qnnl2") pod "700c3c8a-8660-43c6-8fe9-d54e31ac6956" (UID: "700c3c8a-8660-43c6-8fe9-d54e31ac6956"). InnerVolumeSpecName "kube-api-access-qnnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:44:47.935824 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:47.935679    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "700c3c8a-8660-43c6-8fe9-d54e31ac6956" (UID: "700c3c8a-8660-43c6-8fe9-d54e31ac6956"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:44:48.034170 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.034135    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/700c3c8a-8660-43c6-8fe9-d54e31ac6956-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:44:48.034364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.034195    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/700c3c8a-8660-43c6-8fe9-d54e31ac6956-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:44:48.034364 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.034212    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnl2\" (UniqueName: \"kubernetes.io/projected/700c3c8a-8660-43c6-8fe9-d54e31ac6956-kube-api-access-qnnl2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:44:48.648601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.648500    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn" event={"ID":"700c3c8a-8660-43c6-8fe9-d54e31ac6956","Type":"ContainerDied","Data":"2461b5886ccc462f31cefb1024efb6bd2f2e8ccc3b55f6f7221f3d83b0a37baa"}
Jul 27 20:44:48.648601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.648579    2531 scope.go:117] "RemoveContainer" containerID="5515c17aa40be2ca60289b1c5e6b93a2a2a246f13a89695739986307a75eb48c"
Jul 27 20:44:48.648601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.648522    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"
Jul 27 20:44:48.665759 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.665735    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:44:48.667995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:48.667973    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a12508-65478c45-bg2vn"]
Jul 27 20:44:50.340864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:50.340833    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" path="/var/lib/kubelet/pods/700c3c8a-8660-43c6-8fe9-d54e31ac6956/volumes"
Jul 27 20:44:52.108088 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:52.108042    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"]
Jul 27 20:44:52.110638 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:52.110610    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a12508-mz9lw"]
Jul 27 20:44:52.340598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:52.340567    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7da04c2c-c777-4221-b857-81c7387af737" path="/var/lib/kubelet/pods/7da04c2c-c777-4221-b857-81c7387af737/volumes"
Jul 27 20:44:59.101034 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.100999    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"]
Jul 27 20:44:59.101710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101683    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" containerName="maas-api"
Jul 27 20:44:59.101710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101710    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" containerName="maas-api"
Jul 27 20:44:59.101900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101728    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7da04c2c-c777-4221-b857-81c7387af737" containerName="revoke-keys"
Jul 27 20:44:59.101900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101734    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="7da04c2c-c777-4221-b857-81c7387af737" containerName="revoke-keys"
Jul 27 20:44:59.101900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101837    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="700c3c8a-8660-43c6-8fe9-d54e31ac6956" containerName="maas-api"
Jul 27 20:44:59.101900 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.101848    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="7da04c2c-c777-4221-b857-81c7387af737" containerName="revoke-keys"
Jul 27 20:44:59.106842 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.106826    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:44:59.109324 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.109292    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:44:59.113163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.113135    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"]
Jul 27 20:44:59.140250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.140212    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46\" (UID: \"83c04359-2739-4011-a0d9-bb1360a71856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:44:59.241659 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.241622    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46\" (UID: \"83c04359-2739-4011-a0d9-bb1360a71856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:44:59.242268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.242249    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46\" (UID: \"83c04359-2739-4011-a0d9-bb1360a71856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:44:59.419716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.419636    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:44:59.560871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.560839    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"]
Jul 27 20:44:59.562386 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:44:59.562359    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c04359_2739_4011_a0d9_bb1360a71856.slice/crio-c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6 WatchSource:0}: Error finding container c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6: Status 404 returned error can't find the container with id c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6
Jul 27 20:44:59.724685 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:44:59.724596    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46" event={"ID":"83c04359-2739-4011-a0d9-bb1360a71856","Type":"ContainerStarted","Data":"c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6"}
Jul 27 20:45:00.141262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.141158    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"]
Jul 27 20:45:00.145261 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.145231    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"]
Jul 27 20:45:00.145388 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.145356    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"
Jul 27 20:45:00.149162 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.149139    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"
Jul 27 20:45:00.151734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.151436    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"]
Jul 27 20:45:00.154858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.154829    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"]
Jul 27 20:45:00.161408 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.161387    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"
Jul 27 20:45:00.168312 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.168294    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"
Jul 27 20:45:00.316704 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.316667    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"]
Jul 27 20:45:00.333166 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:45:00.333129    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc790a0a2_5c2b_4a04_aaa6_207ca43588f7.slice/crio-3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15 WatchSource:0}: Error finding container 3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15: Status 404 returned error can't find the container with id 3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15
Jul 27 20:45:00.344956 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.344732    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"]
Jul 27 20:45:00.352703 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:45:00.352673    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2914286b_7b50_4192_aaed_1e7e678b96f3.slice/crio-581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072 WatchSource:0}: Error finding container 581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072: Status 404 returned error can't find the container with id 581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072
Jul 27 20:45:00.732698 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.732549    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m" event={"ID":"c790a0a2-5c2b-4a04-aaa6-207ca43588f7","Type":"ContainerStarted","Data":"9f5e186d83d55c2851f815093e81ae1ebb5835fd9dc24de53fdf73bc86fc81bf"}
Jul 27 20:45:00.732698 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.732622    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m" event={"ID":"c790a0a2-5c2b-4a04-aaa6-207ca43588f7","Type":"ContainerStarted","Data":"3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15"}
Jul 27 20:45:00.734450 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.734348    2531 generic.go:356] "Generic (PLEG): container finished" podID="83c04359-2739-4011-a0d9-bb1360a71856" containerID="f2d65f9a6eebda6635ae1fac5124629e0c7f4d21408625dc09f2d04936abec8b" exitCode=0
Jul 27 20:45:00.734450 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.734412    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46" event={"ID":"83c04359-2739-4011-a0d9-bb1360a71856","Type":"ContainerDied","Data":"f2d65f9a6eebda6635ae1fac5124629e0c7f4d21408625dc09f2d04936abec8b"}
Jul 27 20:45:00.736248 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.736206    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q" event={"ID":"2914286b-7b50-4192-aaed-1e7e678b96f3","Type":"ContainerStarted","Data":"581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072"}
Jul 27 20:45:00.749500 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:00.749451    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m" podStartSLOduration=0.749431473 podStartE2EDuration="749.431473ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:00.747744426 +0000 UTC m=+2466.842329905" watchObservedRunningTime="2026-07-27 20:45:00.749431473 +0000 UTC m=+2466.844016939"
Jul 27 20:45:01.745100 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.745013    2531 generic.go:356] "Generic (PLEG): container finished" podID="c790a0a2-5c2b-4a04-aaa6-207ca43588f7" containerID="9f5e186d83d55c2851f815093e81ae1ebb5835fd9dc24de53fdf73bc86fc81bf" exitCode=0
Jul 27 20:45:01.745547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.745107    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m" event={"ID":"c790a0a2-5c2b-4a04-aaa6-207ca43588f7","Type":"ContainerDied","Data":"9f5e186d83d55c2851f815093e81ae1ebb5835fd9dc24de53fdf73bc86fc81bf"}
Jul 27 20:45:01.746737 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.746711    2531 generic.go:356] "Generic (PLEG): container finished" podID="2914286b-7b50-4192-aaed-1e7e678b96f3" containerID="dc73211b41a67569c8c9617c2ec417d73f01dfec31e3e6f97758eb884a553942" exitCode=0
Jul 27 20:45:01.747157 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.746766    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q" event={"ID":"2914286b-7b50-4192-aaed-1e7e678b96f3","Type":"ContainerDied","Data":"dc73211b41a67569c8c9617c2ec417d73f01dfec31e3e6f97758eb884a553942"}
Jul 27 20:45:01.890843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.890815    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:45:01.964853 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.964817    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca\") pod \"83c04359-2739-4011-a0d9-bb1360a71856\" (UID: \"83c04359-2739-4011-a0d9-bb1360a71856\") "
Jul 27 20:45:01.965168 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:01.965144    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83c04359-2739-4011-a0d9-bb1360a71856" (UID: "83c04359-2739-4011-a0d9-bb1360a71856"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:02.065943 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.065909    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c04359-2739-4011-a0d9-bb1360a71856-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:45:02.754213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.754110    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"
Jul 27 20:45:02.754213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.754147    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46" event={"ID":"83c04359-2739-4011-a0d9-bb1360a71856","Type":"ContainerDied","Data":"c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6"}
Jul 27 20:45:02.754701 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.754229    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3bae0eeba930359aea6cd9798f66cf22b27452f4854a65f9d96051b92075db6"
Jul 27 20:45:02.900930 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.900894    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"
Jul 27 20:45:02.932992 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:02.932958    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"
Jul 27 20:45:03.762216 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.762167    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"
Jul 27 20:45:03.762614 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.762217    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m" event={"ID":"c790a0a2-5c2b-4a04-aaa6-207ca43588f7","Type":"ContainerDied","Data":"3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15"}
Jul 27 20:45:03.762614 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.762270    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc4ca449345712c9aea44ee2ea8edc9c8342fe099f4830dfe059b1740351e15"
Jul 27 20:45:03.763524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.763498    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q" event={"ID":"2914286b-7b50-4192-aaed-1e7e678b96f3","Type":"ContainerDied","Data":"581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072"}
Jul 27 20:45:03.763524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.763521    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753085-v4q6q"
Jul 27 20:45:03.763694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:03.763536    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581a896ed93b0e10fb2d0d5653725586f1ce4cbf9ad3b5e4b64b970ac08ae072"
Jul 27 20:45:09.516710 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.516669    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:45:09.517275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.516943    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" podUID="483864b3-631c-455c-8b0d-53a8f60ebf3b" containerName="maas-api" containerID="cri-o://866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83" gracePeriod=30
Jul 27 20:45:09.532802 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.532777    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"]
Jul 27 20:45:09.541507 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.541482    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-eb7a86-29753085-dzq2m"]
Jul 27 20:45:09.778793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.778768    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:45:09.799832 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.799805    2531 generic.go:356] "Generic (PLEG): container finished" podID="483864b3-631c-455c-8b0d-53a8f60ebf3b" containerID="866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83" exitCode=0
Jul 27 20:45:09.799984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.799867    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"
Jul 27 20:45:09.799984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.799903    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" event={"ID":"483864b3-631c-455c-8b0d-53a8f60ebf3b","Type":"ContainerDied","Data":"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"}
Jul 27 20:45:09.799984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.799949    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw" event={"ID":"483864b3-631c-455c-8b0d-53a8f60ebf3b","Type":"ContainerDied","Data":"468c42ea36e48c8d263047e471cdc1ffd76c4c764bb71c5ff8bbbdaed2112c9c"}
Jul 27 20:45:09.799984 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.799979    2531 scope.go:117] "RemoveContainer" containerID="866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"
Jul 27 20:45:09.810719 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.810687    2531 scope.go:117] "RemoveContainer" containerID="866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"
Jul 27 20:45:09.811003 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:45:09.810979    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83\": container with ID starting with 866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83 not found: ID does not exist" containerID="866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"
Jul 27 20:45:09.811078 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.811016    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83"} err="failed to get container status \"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83\": rpc error: code = NotFound desc = could not find container \"866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83\": container with ID starting with 866007cb68c9189c163d56727bc4b79ce1642b1d0b2e0232d8f6b5ce46fe6c83 not found: ID does not exist"
Jul 27 20:45:09.838844 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.838822    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") pod \"483864b3-631c-455c-8b0d-53a8f60ebf3b\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") "
Jul 27 20:45:09.838975 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.838912    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca\") pod \"483864b3-631c-455c-8b0d-53a8f60ebf3b\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") "
Jul 27 20:45:09.839039 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.838975    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlscz\" (UniqueName: \"kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz\") pod \"483864b3-631c-455c-8b0d-53a8f60ebf3b\" (UID: \"483864b3-631c-455c-8b0d-53a8f60ebf3b\") "
Jul 27 20:45:09.839314 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.839289    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "483864b3-631c-455c-8b0d-53a8f60ebf3b" (UID: "483864b3-631c-455c-8b0d-53a8f60ebf3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:09.840946 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.840920    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "483864b3-631c-455c-8b0d-53a8f60ebf3b" (UID: "483864b3-631c-455c-8b0d-53a8f60ebf3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:45:09.841037 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.841011    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz" (OuterVolumeSpecName: "kube-api-access-jlscz") pod "483864b3-631c-455c-8b0d-53a8f60ebf3b" (UID: "483864b3-631c-455c-8b0d-53a8f60ebf3b"). InnerVolumeSpecName "kube-api-access-jlscz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:45:09.940522 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.940489    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlscz\" (UniqueName: \"kubernetes.io/projected/483864b3-631c-455c-8b0d-53a8f60ebf3b-kube-api-access-jlscz\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:45:09.940522 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.940518    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/483864b3-631c-455c-8b0d-53a8f60ebf3b-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:45:09.940522 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:09.940528    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/483864b3-631c-455c-8b0d-53a8f60ebf3b-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:45:10.125469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:10.125438    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:45:10.131264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:10.131238    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb7a86-7bcb987d7b-6kkkw"]
Jul 27 20:45:10.340216 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:10.340168    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483864b3-631c-455c-8b0d-53a8f60ebf3b" path="/var/lib/kubelet/pods/483864b3-631c-455c-8b0d-53a8f60ebf3b/volumes"
Jul 27 20:45:10.340624 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:10.340596    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c790a0a2-5c2b-4a04-aaa6-207ca43588f7" path="/var/lib/kubelet/pods/c790a0a2-5c2b-4a04-aaa6-207ca43588f7/volumes"
Jul 27 20:45:14.175716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:14.175678    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"]
Jul 27 20:45:14.177883 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:14.177855    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eb7a86-q9f46"]
Jul 27 20:45:14.339279 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:14.339250    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c04359-2739-4011-a0d9-bb1360a71856" path="/var/lib/kubelet/pods/83c04359-2739-4011-a0d9-bb1360a71856/volumes"
Jul 27 20:45:24.816190 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816132    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:45:24.816703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816679    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c04359-2739-4011-a0d9-bb1360a71856" containerName="revoke-keys"
Jul 27 20:45:24.816703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816704    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c04359-2739-4011-a0d9-bb1360a71856" containerName="revoke-keys"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816714    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2914286b-7b50-4192-aaed-1e7e678b96f3" containerName="cleanup"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816721    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="2914286b-7b50-4192-aaed-1e7e678b96f3" containerName="cleanup"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816738    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483864b3-631c-455c-8b0d-53a8f60ebf3b" containerName="maas-api"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816747    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="483864b3-631c-455c-8b0d-53a8f60ebf3b" containerName="maas-api"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816763    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c790a0a2-5c2b-4a04-aaa6-207ca43588f7" containerName="cleanup"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816769    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="c790a0a2-5c2b-4a04-aaa6-207ca43588f7" containerName="cleanup"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816837    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="c790a0a2-5c2b-4a04-aaa6-207ca43588f7" containerName="cleanup"
Jul 27 20:45:24.816843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816844    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c04359-2739-4011-a0d9-bb1360a71856" containerName="revoke-keys"
Jul 27 20:45:24.817287 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816857    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="483864b3-631c-455c-8b0d-53a8f60ebf3b" containerName="maas-api"
Jul 27 20:45:24.817287 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.816867    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="2914286b-7b50-4192-aaed-1e7e678b96f3" containerName="cleanup"
Jul 27 20:45:24.821284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.821264    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.823553 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.823529    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3f4799-serving-cert\""
Jul 27 20:45:24.830793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.830768    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:45:24.881385 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.881348    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnwm\" (UniqueName: \"kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.881385 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.881387    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.881657 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.881526    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.983167 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.983121    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnwm\" (UniqueName: \"kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.983167 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.983165    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.983439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.983298    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.983506 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:45:24.983440    2531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-serving-cert: secret "maas-api-e2e-api-a-3f4799-serving-cert" not found
Jul 27 20:45:24.983566 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:45:24.983521    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls podName:2aa942f6-82b2-4dcb-a56f-9881abe0df80 nodeName:}" failed. No retries permitted until 2026-07-27 20:45:25.483503953 +0000 UTC m=+2491.578089396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls") pod "maas-api-e2e-api-a-3f4799-6665c9849-244fr" (UID: "2aa942f6-82b2-4dcb-a56f-9881abe0df80") : secret "maas-api-e2e-api-a-3f4799-serving-cert" not found
Jul 27 20:45:24.983924 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.983901    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:24.991846 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:24.991824    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnwm\" (UniqueName: \"kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:25.489616 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:25.489575    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:25.492243 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:25.492213    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") pod \"maas-api-e2e-api-a-3f4799-6665c9849-244fr\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:25.736254 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:25.736219    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:25.881726 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:25.881699    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:45:25.882747 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:45:25.882725    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa942f6_82b2_4dcb_a56f_9881abe0df80.slice/crio-93f47a962ba5b128204970bee51ab4f007aefdff410404a6be6247e25423b958 WatchSource:0}: Error finding container 93f47a962ba5b128204970bee51ab4f007aefdff410404a6be6247e25423b958: Status 404 returned error can't find the container with id 93f47a962ba5b128204970bee51ab4f007aefdff410404a6be6247e25423b958
Jul 27 20:45:25.900700 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:25.900672    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" event={"ID":"2aa942f6-82b2-4dcb-a56f-9881abe0df80","Type":"ContainerStarted","Data":"93f47a962ba5b128204970bee51ab4f007aefdff410404a6be6247e25423b958"}
Jul 27 20:45:26.908239 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:26.908200    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" event={"ID":"2aa942f6-82b2-4dcb-a56f-9881abe0df80","Type":"ContainerStarted","Data":"6c3156cd961928b43da9c161c1b3687720a6d4d7a8416dd36f50087fe4b4cc73"}
Jul 27 20:45:26.908852 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:26.908278    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:26.926195 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:26.926125    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" podStartSLOduration=2.075103333 podStartE2EDuration="2.926113736s" totalImagesPullingTime="851.010403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:26.924444583 +0000 UTC m=+2493.019030059" watchObservedRunningTime="2026-07-27 20:45:26.926113736 +0000 UTC m=+2493.020699199"
Jul 27 20:45:32.808275 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.808239    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:45:32.812902 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.812878    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:32.816914 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.816893    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5a7e30-serving-cert\""
Jul 27 20:45:32.826244 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.826222    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:45:32.921163 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.921130    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:45:32.973520 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.973485    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:32.973693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.973606    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzhh\" (UniqueName: \"kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:32.973693 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:32.973656    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.074703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.074601    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzhh\" (UniqueName: \"kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.074703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.074647    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.074959 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.074731    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.074959 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:45:33.074838    2531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-serving-cert: secret "maas-api-e2e-api-b-5a7e30-serving-cert" not found
Jul 27 20:45:33.074959 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:45:33.074928    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls podName:02ca3cc8-60b6-4901-964f-1c0b1494aea3 nodeName:}" failed. No retries permitted until 2026-07-27 20:45:33.574908579 +0000 UTC m=+2499.669494023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls") pod "maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" (UID: "02ca3cc8-60b6-4901-964f-1c0b1494aea3") : secret "maas-api-e2e-api-b-5a7e30-serving-cert" not found
Jul 27 20:45:33.075435 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.075412    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.086394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.086368    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzhh\" (UniqueName: \"kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.579993 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.579958    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.582534 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.582504    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") pod \"maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.724894 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.724858    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:33.876196 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.876156    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:45:33.878605 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:45:33.878566    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ca3cc8_60b6_4901_964f_1c0b1494aea3.slice/crio-d88ad753b31fc4aee8bc6596f340989e022e7cb68f2cccf878d4cdc0b6708b65 WatchSource:0}: Error finding container d88ad753b31fc4aee8bc6596f340989e022e7cb68f2cccf878d4cdc0b6708b65: Status 404 returned error can't find the container with id d88ad753b31fc4aee8bc6596f340989e022e7cb68f2cccf878d4cdc0b6708b65
Jul 27 20:45:33.955120 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:33.955088    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" event={"ID":"02ca3cc8-60b6-4901-964f-1c0b1494aea3","Type":"ContainerStarted","Data":"d88ad753b31fc4aee8bc6596f340989e022e7cb68f2cccf878d4cdc0b6708b65"}
Jul 27 20:45:34.963330 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:34.963286    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" event={"ID":"02ca3cc8-60b6-4901-964f-1c0b1494aea3","Type":"ContainerStarted","Data":"0bdaa2c968f20366a8c34103830779dc2ba94cf865a445c982399da489b3a55f"}
Jul 27 20:45:34.963736 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:34.963456    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:34.981598 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:34.981549    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" podStartSLOduration=2.586526469 podStartE2EDuration="2.981535526s" totalImagesPullingTime="395.009057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:34.978986955 +0000 UTC m=+2501.073572418" watchObservedRunningTime="2026-07-27 20:45:34.981535526 +0000 UTC m=+2501.076120990"
Jul 27 20:45:40.976601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:40.976561    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:45:45.154913 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.154875    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"]
Jul 27 20:45:45.158929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.158912    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:45.160969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.160941    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:45:45.165342 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.165295    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"]
Jul 27 20:45:45.186592 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.186565    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f4799-495kb\" (UID: \"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:45.287565 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.287523    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f4799-495kb\" (UID: \"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:45.288221 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.288197    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3f4799-495kb\" (UID: \"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:45.472357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.472280    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:45.613222 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:45.613188    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"]
Jul 27 20:45:45.614225 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:45:45.614167    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb17db2_5c4a_41ba_923c_7ea2c2c0fe90.slice/crio-addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248 WatchSource:0}: Error finding container addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248: Status 404 returned error can't find the container with id addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248
Jul 27 20:45:46.029103 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:46.029066    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb" event={"ID":"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90","Type":"ContainerStarted","Data":"e04fd1aa5b79f4b6f6be62b63d7ba1502c02dd778c44128185bd495d815f3a4c"}
Jul 27 20:45:46.029279 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:46.029119    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb" event={"ID":"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90","Type":"ContainerStarted","Data":"addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248"}
Jul 27 20:45:46.045825 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:46.045759    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb" podStartSLOduration=1.045743692 podStartE2EDuration="1.045743692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:45:46.043363476 +0000 UTC m=+2512.137948942" watchObservedRunningTime="2026-07-27 20:45:46.045743692 +0000 UTC m=+2512.140329157"
Jul 27 20:45:47.037869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:47.037825    2531 generic.go:356] "Generic (PLEG): container finished" podID="bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" containerID="e04fd1aa5b79f4b6f6be62b63d7ba1502c02dd778c44128185bd495d815f3a4c" exitCode=0
Jul 27 20:45:47.038270 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:47.037895    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb" event={"ID":"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90","Type":"ContainerDied","Data":"e04fd1aa5b79f4b6f6be62b63d7ba1502c02dd778c44128185bd495d815f3a4c"}
Jul 27 20:45:48.195873 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:48.195849    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:48.216263 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:48.216229    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca\") pod \"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90\" (UID: \"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90\") "
Jul 27 20:45:48.216805 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:48.216780    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" (UID: "bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:45:48.318035 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:48.317960    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:45:49.053136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:49.053104    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"
Jul 27 20:45:49.053356 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:49.053130    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb" event={"ID":"bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90","Type":"ContainerDied","Data":"addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248"}
Jul 27 20:45:49.053356 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:49.053223    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="addc7fdae85b5e06a6bdd5918459250dce700ddd4cdfa8b2a7adbfb9bd3f9248"
Jul 27 20:45:54.695683 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:45:54.695651    2531 scope.go:117] "RemoveContainer" containerID="8caf9a720f5850a5e4e8f55608369ae1f9e56f9f364b3c7f2e81df2722c549f0"
Jul 27 20:46:00.955711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:00.955679    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:46:00.956523 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:00.955885    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" podUID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" containerName="maas-api" containerID="cri-o://6c3156cd961928b43da9c161c1b3687720a6d4d7a8416dd36f50087fe4b4cc73" gracePeriod=30
Jul 27 20:46:01.126612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.126583    2531 generic.go:356] "Generic (PLEG): container finished" podID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" containerID="6c3156cd961928b43da9c161c1b3687720a6d4d7a8416dd36f50087fe4b4cc73" exitCode=0
Jul 27 20:46:01.126758 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.126656    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" event={"ID":"2aa942f6-82b2-4dcb-a56f-9881abe0df80","Type":"ContainerDied","Data":"6c3156cd961928b43da9c161c1b3687720a6d4d7a8416dd36f50087fe4b4cc73"}
Jul 27 20:46:01.210987 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.210932    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:46:01.348726 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.348688    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") pod \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") "
Jul 27 20:46:01.348932 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.348848    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca\") pod \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") "
Jul 27 20:46:01.348932 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.348874    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdnwm\" (UniqueName: \"kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm\") pod \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\" (UID: \"2aa942f6-82b2-4dcb-a56f-9881abe0df80\") "
Jul 27 20:46:01.349281 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.349251    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2aa942f6-82b2-4dcb-a56f-9881abe0df80" (UID: "2aa942f6-82b2-4dcb-a56f-9881abe0df80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:46:01.351011 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.350964    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2aa942f6-82b2-4dcb-a56f-9881abe0df80" (UID: "2aa942f6-82b2-4dcb-a56f-9881abe0df80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:46:01.351011 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.350988    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm" (OuterVolumeSpecName: "kube-api-access-sdnwm") pod "2aa942f6-82b2-4dcb-a56f-9881abe0df80" (UID: "2aa942f6-82b2-4dcb-a56f-9881abe0df80"). InnerVolumeSpecName "kube-api-access-sdnwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:46:01.449914 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.449878    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa942f6-82b2-4dcb-a56f-9881abe0df80-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:01.449914 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.449906    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa942f6-82b2-4dcb-a56f-9881abe0df80-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:01.449914 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:01.449915    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnwm\" (UniqueName: \"kubernetes.io/projected/2aa942f6-82b2-4dcb-a56f-9881abe0df80-kube-api-access-sdnwm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:02.134393 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.134346    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"
Jul 27 20:46:02.134858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.134363    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr" event={"ID":"2aa942f6-82b2-4dcb-a56f-9881abe0df80","Type":"ContainerDied","Data":"93f47a962ba5b128204970bee51ab4f007aefdff410404a6be6247e25423b958"}
Jul 27 20:46:02.134858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.134534    2531 scope.go:117] "RemoveContainer" containerID="6c3156cd961928b43da9c161c1b3687720a6d4d7a8416dd36f50087fe4b4cc73"
Jul 27 20:46:02.158197 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.158151    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:46:02.161962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.161941    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f4799-6665c9849-244fr"]
Jul 27 20:46:02.340991 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:02.340958    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" path="/var/lib/kubelet/pods/2aa942f6-82b2-4dcb-a56f-9881abe0df80/volumes"
Jul 27 20:46:05.236241 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:05.236203    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"]
Jul 27 20:46:05.241990 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:05.241952    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3f4799-495kb"]
Jul 27 20:46:06.340573 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:06.340536    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" path="/var/lib/kubelet/pods/bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90/volumes"
Jul 27 20:46:12.249797 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.249755    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"]
Jul 27 20:46:12.250553 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250532    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" containerName="maas-api"
Jul 27 20:46:12.250608 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250557    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" containerName="maas-api"
Jul 27 20:46:12.250608 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250582    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" containerName="revoke-keys"
Jul 27 20:46:12.250608 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250590    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" containerName="revoke-keys"
Jul 27 20:46:12.250734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250723    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa942f6-82b2-4dcb-a56f-9881abe0df80" containerName="maas-api"
Jul 27 20:46:12.250771 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.250741    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb17db2-5c4a-41ba-923c-7ea2c2c0fe90" containerName="revoke-keys"
Jul 27 20:46:12.261498 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.261465    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:12.264711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.264593    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:46:12.268008 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.267980    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"]
Jul 27 20:46:12.356976 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.356942    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj\" (UID: \"cb1444aa-7ac5-49b0-a475-570eed355255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:12.458513 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.458474    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj\" (UID: \"cb1444aa-7ac5-49b0-a475-570eed355255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:12.459118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.459091    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj\" (UID: \"cb1444aa-7ac5-49b0-a475-570eed355255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:12.575806 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.575775    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:12.714823 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:12.714796    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"]
Jul 27 20:46:12.717215 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:46:12.717168    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1444aa_7ac5_49b0_a475_570eed355255.slice/crio-abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471 WatchSource:0}: Error finding container abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471: Status 404 returned error can't find the container with id abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471
Jul 27 20:46:13.203477 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:13.203385    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj" event={"ID":"cb1444aa-7ac5-49b0-a475-570eed355255","Type":"ContainerStarted","Data":"8113f7ba86b91d81b8ca57be234989308448e1e8f37b9ab80efeff6272b7ddca"}
Jul 27 20:46:13.203477 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:13.203445    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj" event={"ID":"cb1444aa-7ac5-49b0-a475-570eed355255","Type":"ContainerStarted","Data":"abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471"}
Jul 27 20:46:13.219025 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:13.218978    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj" podStartSLOduration=1.218962434 podStartE2EDuration="1.218962434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:46:13.217607243 +0000 UTC m=+2539.312192718" watchObservedRunningTime="2026-07-27 20:46:13.218962434 +0000 UTC m=+2539.313547897"
Jul 27 20:46:14.211462 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:14.211429    2531 generic.go:356] "Generic (PLEG): container finished" podID="cb1444aa-7ac5-49b0-a475-570eed355255" containerID="8113f7ba86b91d81b8ca57be234989308448e1e8f37b9ab80efeff6272b7ddca" exitCode=0
Jul 27 20:46:14.211932 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:14.211507    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj" event={"ID":"cb1444aa-7ac5-49b0-a475-570eed355255","Type":"ContainerDied","Data":"8113f7ba86b91d81b8ca57be234989308448e1e8f37b9ab80efeff6272b7ddca"}
Jul 27 20:46:15.375479 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:15.375451    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:15.486101 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:15.486024    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca\") pod \"cb1444aa-7ac5-49b0-a475-570eed355255\" (UID: \"cb1444aa-7ac5-49b0-a475-570eed355255\") "
Jul 27 20:46:15.486395 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:15.486374    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb1444aa-7ac5-49b0-a475-570eed355255" (UID: "cb1444aa-7ac5-49b0-a475-570eed355255"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:46:15.587634 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:15.587603    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1444aa-7ac5-49b0-a475-570eed355255-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:16.224690 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:16.224657    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj" event={"ID":"cb1444aa-7ac5-49b0-a475-570eed355255","Type":"ContainerDied","Data":"abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471"}
Jul 27 20:46:16.224864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:16.224704    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abbc661c114e92037810fe91663436175da78e8566c91d6b2549ebf66d74e471"
Jul 27 20:46:16.224864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:16.224702    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"
Jul 27 20:46:28.049563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.049526    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:46:28.050499 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.050463    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" podUID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" containerName="maas-api" containerID="cri-o://0bdaa2c968f20366a8c34103830779dc2ba94cf865a445c982399da489b3a55f" gracePeriod=30
Jul 27 20:46:28.295474 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.295442    2531 generic.go:356] "Generic (PLEG): container finished" podID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" containerID="0bdaa2c968f20366a8c34103830779dc2ba94cf865a445c982399da489b3a55f" exitCode=0
Jul 27 20:46:28.295623 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.295518    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" event={"ID":"02ca3cc8-60b6-4901-964f-1c0b1494aea3","Type":"ContainerDied","Data":"0bdaa2c968f20366a8c34103830779dc2ba94cf865a445c982399da489b3a55f"}
Jul 27 20:46:28.315001 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.314950    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:46:28.402371 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.402329    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlzhh\" (UniqueName: \"kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh\") pod \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") "
Jul 27 20:46:28.402664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.402642    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca\") pod \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") "
Jul 27 20:46:28.402864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.402839    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") pod \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\" (UID: \"02ca3cc8-60b6-4901-964f-1c0b1494aea3\") "
Jul 27 20:46:28.403099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.403031    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02ca3cc8-60b6-4901-964f-1c0b1494aea3" (UID: "02ca3cc8-60b6-4901-964f-1c0b1494aea3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:46:28.403588 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.403552    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ca3cc8-60b6-4901-964f-1c0b1494aea3-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:28.405063 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.405033    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02ca3cc8-60b6-4901-964f-1c0b1494aea3" (UID: "02ca3cc8-60b6-4901-964f-1c0b1494aea3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:46:28.405225 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.405202    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh" (OuterVolumeSpecName: "kube-api-access-vlzhh") pod "02ca3cc8-60b6-4901-964f-1c0b1494aea3" (UID: "02ca3cc8-60b6-4901-964f-1c0b1494aea3"). InnerVolumeSpecName "kube-api-access-vlzhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:46:28.504597 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.504557    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzhh\" (UniqueName: \"kubernetes.io/projected/02ca3cc8-60b6-4901-964f-1c0b1494aea3-kube-api-access-vlzhh\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:28.504597 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:28.504595    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ca3cc8-60b6-4901-964f-1c0b1494aea3-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:46:29.302485 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:29.302455    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"
Jul 27 20:46:29.302957 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:29.302489    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2" event={"ID":"02ca3cc8-60b6-4901-964f-1c0b1494aea3","Type":"ContainerDied","Data":"d88ad753b31fc4aee8bc6596f340989e022e7cb68f2cccf878d4cdc0b6708b65"}
Jul 27 20:46:29.302957 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:29.302568    2531 scope.go:117] "RemoveContainer" containerID="0bdaa2c968f20366a8c34103830779dc2ba94cf865a445c982399da489b3a55f"
Jul 27 20:46:29.325713 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:29.325689    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:46:29.329345 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:29.329325    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5a7e30-6d5b9bb5d9-rgvn2"]
Jul 27 20:46:30.340488 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:30.340446    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" path="/var/lib/kubelet/pods/02ca3cc8-60b6-4901-964f-1c0b1494aea3/volumes"
Jul 27 20:46:32.341926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:32.341896    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"]
Jul 27 20:46:32.341926 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:32.341932    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5a7e30-km5xj"]
Jul 27 20:46:34.339738 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:34.339704    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb1444aa-7ac5-49b0-a475-570eed355255" path="/var/lib/kubelet/pods/cb1444aa-7ac5-49b0-a475-570eed355255/volumes"
Jul 27 20:46:43.025440 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.025401    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:46:43.026186 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026155    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" containerName="maas-api"
Jul 27 20:46:43.026237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026187    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" containerName="maas-api"
Jul 27 20:46:43.026237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026207    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb1444aa-7ac5-49b0-a475-570eed355255" containerName="revoke-keys"
Jul 27 20:46:43.026237 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026215    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb1444aa-7ac5-49b0-a475-570eed355255" containerName="revoke-keys"
Jul 27 20:46:43.026328 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026312    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ca3cc8-60b6-4901-964f-1c0b1494aea3" containerName="maas-api"
Jul 27 20:46:43.026369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.026328    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb1444aa-7ac5-49b0-a475-570eed355255" containerName="revoke-keys"
Jul 27 20:46:43.030190 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.030148    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.032748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.032725    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4af3f5-serving-cert\""
Jul 27 20:46:43.040001 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.039977    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:46:43.148888 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.148847    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.149066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.148963    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwpx\" (UniqueName: \"kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.149066 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.149010    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.250401 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.250368    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwpx\" (UniqueName: \"kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.250563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.250411    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.250563 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.250477    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.251141 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.251117    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.252886 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.252862    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.277109 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.277050    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwpx\" (UniqueName: \"kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx\") pod \"maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.345252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.345223    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:43.484287 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:43.484264    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:46:43.486625 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:46:43.486590    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5d49ef_2898_455c_8f77_fea0451058a1.slice/crio-293a364918a0d8cfe205331d1c6a7b9c6103c4bd67a9126637734208e7037b38 WatchSource:0}: Error finding container 293a364918a0d8cfe205331d1c6a7b9c6103c4bd67a9126637734208e7037b38: Status 404 returned error can't find the container with id 293a364918a0d8cfe205331d1c6a7b9c6103c4bd67a9126637734208e7037b38
Jul 27 20:46:44.393215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:44.393161    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" event={"ID":"9f5d49ef-2898-455c-8f77-fea0451058a1","Type":"ContainerStarted","Data":"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"}
Jul 27 20:46:44.393594 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:44.393230    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:46:44.393594 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:44.393249    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" event={"ID":"9f5d49ef-2898-455c-8f77-fea0451058a1","Type":"ContainerStarted","Data":"293a364918a0d8cfe205331d1c6a7b9c6103c4bd67a9126637734208e7037b38"}
Jul 27 20:46:44.413039 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:44.412987    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" podStartSLOduration=0.941579389 podStartE2EDuration="1.412970977s" totalImagesPullingTime="471.391588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:46:44.41185151 +0000 UTC m=+2570.506437001" watchObservedRunningTime="2026-07-27 20:46:44.412970977 +0000 UTC m=+2570.507556442"
Jul 27 20:46:50.404455 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:46:50.404415    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:47:47.107331 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.107284    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:47:47.113410 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.113368    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.125046 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.125002    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"default-dockercfg-dhw8v\""
Jul 27 20:47:47.126825 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.126305    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"kube-root-ca.crt\""
Jul 27 20:47:47.126825 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.126563    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"openshift-service-ca.crt\""
Jul 27 20:47:47.127357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.127338    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"auth-test-model-21ded0-kserve-self-signed-certs\""
Jul 27 20:47:47.132421 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.132084    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:47:47.175697 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.175666    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.175697 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.175698    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.175944 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.175729    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.175944 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.175852    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6bp\" (UniqueName: \"kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.176051 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.176026    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.176106 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.176065    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277486    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277531    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277776 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277561    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277776 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277670    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277776 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277743    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277931 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277801    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6bp\" (UniqueName: \"kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.277986 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277949    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.278040 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.277995    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.278093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.278049    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.279910 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.279888    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.280085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.280068    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.285891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.285856    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6bp\" (UniqueName: \"kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp\") pod \"auth-test-model-21ded0-kserve-549b48ffbc-5w925\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") " pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.438912 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.438824    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:47.590442 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.590408    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:47:47.594207 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:47:47.594146    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb329b4_54d2_4c97_af01_1045c5b9f89c.slice/crio-3f9462507e1d4304098c85afc0e9c7fc2ef4244ec904cbd7344ee7492af8f906 WatchSource:0}: Error finding container 3f9462507e1d4304098c85afc0e9c7fc2ef4244ec904cbd7344ee7492af8f906: Status 404 returned error can't find the container with id 3f9462507e1d4304098c85afc0e9c7fc2ef4244ec904cbd7344ee7492af8f906
Jul 27 20:47:47.772621 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.772576    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" event={"ID":"feb329b4-54d2-4c97-af01-1045c5b9f89c","Type":"ContainerStarted","Data":"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"}
Jul 27 20:47:47.772774 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.772630    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" event={"ID":"feb329b4-54d2-4c97-af01-1045c5b9f89c","Type":"ContainerStarted","Data":"3f9462507e1d4304098c85afc0e9c7fc2ef4244ec904cbd7344ee7492af8f906"}
Jul 27 20:47:47.789455 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:47.789408    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" podStartSLOduration=0.78939352 podStartE2EDuration="789.39352ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:47:47.787416837 +0000 UTC m=+2633.882002301" watchObservedRunningTime="2026-07-27 20:47:47.78939352 +0000 UTC m=+2633.883978984"
Jul 27 20:47:54.749273 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:54.749235    2531 scope.go:117] "RemoveContainer" containerID="f132d140ca484a00dcea6a6398c955b1019c86f702a7919a60fa391363dce26f"
Jul 27 20:47:57.439365 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:57.439326    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:57.439365 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:57.439380    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:57.451830 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:57.451805    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:47:57.840301 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:47:57.840271    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:48:54.576032 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:48:54.575947    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:48:54.587079 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:48:54.587058    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:50:54.792394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:50:54.792358    2531 scope.go:117] "RemoveContainer" containerID="524ec8d61ea77b5bccbbe37729faa963213e4214789b5a87be2337fb9224348a"
Jul 27 20:51:54.814398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:51:54.814325    2531 scope.go:117] "RemoveContainer" containerID="e04fd1aa5b79f4b6f6be62b63d7ba1502c02dd778c44128185bd495d815f3a4c"
Jul 27 20:51:54.823245 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:51:54.823225    2531 scope.go:117] "RemoveContainer" containerID="9f5e186d83d55c2851f815093e81ae1ebb5835fd9dc24de53fdf73bc86fc81bf"
Jul 27 20:51:54.831284 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:51:54.831263    2531 scope.go:117] "RemoveContainer" containerID="f2d65f9a6eebda6635ae1fac5124629e0c7f4d21408625dc09f2d04936abec8b"
Jul 27 20:52:07.775137 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:07.775098    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:52:22.794856 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:22.794814    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" podUID="feb329b4-54d2-4c97-af01-1045c5b9f89c" containerName="main" containerID="cri-o://a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d" gracePeriod=45
Jul 27 20:52:23.061836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.061798    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:52:23.092850 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.092815    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093007 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.092941    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093007 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.092979    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn6bp\" (UniqueName: \"kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093091 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093043    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093091 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093066    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home" (OuterVolumeSpecName: "home") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:52:23.093091 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093072    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093141    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs\") pod \"feb329b4-54d2-4c97-af01-1045c5b9f89c\" (UID: \"feb329b4-54d2-4c97-af01-1045c5b9f89c\") "
Jul 27 20:52:23.093269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093235    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:52:23.093380 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093361    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache" (OuterVolumeSpecName: "model-cache") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:52:23.093583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093550    2531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.093583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093576    2531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.093583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.093588    2531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.095464 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.095436    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:52:23.095638 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.095616    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm" (OuterVolumeSpecName: "dshm") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:52:23.095723 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.095664    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp" (OuterVolumeSpecName: "kube-api-access-cn6bp") pod "feb329b4-54d2-4c97-af01-1045c5b9f89c" (UID: "feb329b4-54d2-4c97-af01-1045c5b9f89c"). InnerVolumeSpecName "kube-api-access-cn6bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:52:23.194126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.194092    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6bp\" (UniqueName: \"kubernetes.io/projected/feb329b4-54d2-4c97-af01-1045c5b9f89c-kube-api-access-cn6bp\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.194126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.194122    2531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feb329b4-54d2-4c97-af01-1045c5b9f89c-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.194126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.194132    2531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb329b4-54d2-4c97-af01-1045c5b9f89c-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:52:23.397903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.397767    2531 generic.go:356] "Generic (PLEG): container finished" podID="feb329b4-54d2-4c97-af01-1045c5b9f89c" containerID="a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d" exitCode=0
Jul 27 20:52:23.397903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.397842    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" event={"ID":"feb329b4-54d2-4c97-af01-1045c5b9f89c","Type":"ContainerDied","Data":"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"}
Jul 27 20:52:23.397903 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.397885    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925" event={"ID":"feb329b4-54d2-4c97-af01-1045c5b9f89c","Type":"ContainerDied","Data":"3f9462507e1d4304098c85afc0e9c7fc2ef4244ec904cbd7344ee7492af8f906"}
Jul 27 20:52:23.398201 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.397910    2531 scope.go:117] "RemoveContainer" containerID="a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"
Jul 27 20:52:23.398201 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.397821    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"
Jul 27 20:52:23.408323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.408306    2531 scope.go:117] "RemoveContainer" containerID="a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"
Jul 27 20:52:23.408575 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:52:23.408556    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d\": container with ID starting with a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d not found: ID does not exist" containerID="a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"
Jul 27 20:52:23.408643 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.408595    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d"} err="failed to get container status \"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d\": rpc error: code = NotFound desc = could not find container \"a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d\": container with ID starting with a8b5f57bb5a25b3045a51ae1890a9e32adf9dcae67bbd6e387f42d5c7d0c0b9d not found: ID does not exist"
Jul 27 20:52:23.426317 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.426167    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:52:23.429915 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:23.429892    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/auth-test-model-21ded0-kserve-549b48ffbc-5w925"]
Jul 27 20:52:24.339893 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:24.339859    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb329b4-54d2-4c97-af01-1045c5b9f89c" path="/var/lib/kubelet/pods/feb329b4-54d2-4c97-af01-1045c5b9f89c/volumes"
Jul 27 20:52:41.793891 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.793856    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:52:41.794375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.794361    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb329b4-54d2-4c97-af01-1045c5b9f89c" containerName="main"
Jul 27 20:52:41.794375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.794377    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb329b4-54d2-4c97-af01-1045c5b9f89c" containerName="main"
Jul 27 20:52:41.794462 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.794456    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb329b4-54d2-4c97-af01-1045c5b9f89c" containerName="main"
Jul 27 20:52:41.797968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.797951    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.800439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.800413    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"openshift-service-ca.crt\""
Jul 27 20:52:41.800548 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.800488    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"sub-test-model-21ded0-kserve-self-signed-certs\""
Jul 27 20:52:41.800548 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.800499    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"kube-root-ca.crt\""
Jul 27 20:52:41.800548 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.800532    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"default-dockercfg-dhw8v\""
Jul 27 20:52:41.806610 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.806383    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:52:41.892515 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892471    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.892672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892528    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.892672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892598    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvsr\" (UniqueName: \"kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.892764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892718    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.892764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892758    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.892833 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.892783    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.993952 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.993918    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.993952 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.993958    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994219 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.993989    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994219 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994020    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994219 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994037    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994219 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994072    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvsr\" (UniqueName: \"kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994463 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994443    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994546 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994510    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.994601 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.994570    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.996333 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.996315    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:41.996677 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:41.996656    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:42.003323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.003301    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvsr\" (UniqueName: \"kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr\") pod \"sub-test-model-21ded0-kserve-776895f787-gx2jk\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") " pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:42.110283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.110153    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:42.250683 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.250652    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:52:42.252891 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:52:42.252865    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfed9d9e9_0771_40fe_bb8e_e1c3b32db45a.slice/crio-53bcc8b679ef90120fd3619ee5b0a23543a491d33c803c89ebad9eabaf2473ee WatchSource:0}: Error finding container 53bcc8b679ef90120fd3619ee5b0a23543a491d33c803c89ebad9eabaf2473ee: Status 404 returned error can't find the container with id 53bcc8b679ef90120fd3619ee5b0a23543a491d33c803c89ebad9eabaf2473ee
Jul 27 20:52:42.507669 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.507575    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" event={"ID":"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a","Type":"ContainerStarted","Data":"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"}
Jul 27 20:52:42.507669 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.507617    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" event={"ID":"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a","Type":"ContainerStarted","Data":"53bcc8b679ef90120fd3619ee5b0a23543a491d33c803c89ebad9eabaf2473ee"}
Jul 27 20:52:42.526672 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:42.526619    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" podStartSLOduration=1.5266035470000001 podStartE2EDuration="1.526603547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:42.524683982 +0000 UTC m=+2928.619269448" watchObservedRunningTime="2026-07-27 20:52:42.526603547 +0000 UTC m=+2928.621189012"
Jul 27 20:52:52.110976 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:52.110938    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:52.111434 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:52.111265    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:52.124018 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:52.123993    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:52.575797 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:52.575757    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:52:54.861778 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:52:54.861742    2531 scope.go:117] "RemoveContainer" containerID="8113f7ba86b91d81b8ca57be234989308448e1e8f37b9ab80efeff6272b7ddca"
Jul 27 20:53:54.614673 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:53:54.614625    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:53:54.627570 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:53:54.627519    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:53:56.400205 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:53:56.398631    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:54:11.422304 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.422259    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" podUID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" containerName="main" containerID="cri-o://c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b" gracePeriod=45
Jul 27 20:54:11.683472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.683413    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:54:11.779547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779509    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779564    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779648    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pvsr\" (UniqueName: \"kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779673    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779714 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779698    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779931 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779728    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm\") pod \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\" (UID: \"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a\") "
Jul 27 20:54:11.779931 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779826    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:11.779931 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779878    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:11.780091 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.779984    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home" (OuterVolumeSpecName: "home") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:11.780147 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.780114    2531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:11.780147 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.780135    2531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:11.780296 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.780150    2531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:11.781932 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.781910    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm" (OuterVolumeSpecName: "dshm") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:11.782031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.781962    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr" (OuterVolumeSpecName: "kube-api-access-7pvsr") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "kube-api-access-7pvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:11.782085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.782025    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" (UID: "fed9d9e9-0771-40fe-bb8e-e1c3b32db45a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:11.881166 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.881129    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvsr\" (UniqueName: \"kubernetes.io/projected/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-kube-api-access-7pvsr\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:11.881341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.881165    2531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:11.881341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:11.881204    2531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:54:12.010213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.010112    2531 generic.go:356] "Generic (PLEG): container finished" podID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" containerID="c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b" exitCode=0
Jul 27 20:54:12.010213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.010159    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"
Jul 27 20:54:12.010412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.010213    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" event={"ID":"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a","Type":"ContainerDied","Data":"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"}
Jul 27 20:54:12.010412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.010256    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk" event={"ID":"fed9d9e9-0771-40fe-bb8e-e1c3b32db45a","Type":"ContainerDied","Data":"53bcc8b679ef90120fd3619ee5b0a23543a491d33c803c89ebad9eabaf2473ee"}
Jul 27 20:54:12.010412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.010279    2531 scope.go:117] "RemoveContainer" containerID="c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"
Jul 27 20:54:12.020329 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.020308    2531 scope.go:117] "RemoveContainer" containerID="c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"
Jul 27 20:54:12.020643 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:54:12.020591    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b\": container with ID starting with c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b not found: ID does not exist" containerID="c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"
Jul 27 20:54:12.020742 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.020645    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b"} err="failed to get container status \"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b\": rpc error: code = NotFound desc = could not find container \"c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b\": container with ID starting with c465a6354deabbafda3dfd30c57c0054ea129d767c8c1683582bfbad63d7182b not found: ID does not exist"
Jul 27 20:54:12.033654 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.033631    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:54:12.037382 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.037358    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/sub-test-model-21ded0-kserve-776895f787-gx2jk"]
Jul 27 20:54:12.339864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:12.339830    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" path="/var/lib/kubelet/pods/fed9d9e9-0771-40fe-bb8e-e1c3b32db45a/volumes"
Jul 27 20:54:20.202559 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.202406    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:54:20.203150 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.203130    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" containerName="main"
Jul 27 20:54:20.203280 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.203152    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" containerName="main"
Jul 27 20:54:20.203353 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.203337    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed9d9e9-0771-40fe-bb8e-e1c3b32db45a" containerName="main"
Jul 27 20:54:20.207348 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.207329    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.209756 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.209735    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"openshift-service-ca.crt\""
Jul 27 20:54:20.209867 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.209834    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"kube-root-ca.crt\""
Jul 27 20:54:20.210617 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.210597    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"rate-test-model-21ded0-kserve-self-signed-certs\""
Jul 27 20:54:20.210725 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.210641    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-21ded0\"/\"default-dockercfg-dhw8v\""
Jul 27 20:54:20.215793 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.215773    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:54:20.264547 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264511    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.264711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264674    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.264711 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264700    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgr7p\" (UniqueName: \"kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.264803 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264735    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.264803 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264768    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.264869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.264808    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.365991    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366032    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366290 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366184    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366290 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366235    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366378    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366448 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366409    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgr7p\" (UniqueName: \"kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366440    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366477    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.366716 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.366695    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.368361 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.368342    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.368469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.368452    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.378669 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.378644    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgr7p\" (UniqueName: \"kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p\") pod \"rate-test-model-21ded0-kserve-655cf5cc55-7n955\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") " pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.517423 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.517402    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:20.661276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:20.661248    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:54:20.663107 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:54:20.663069    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8ee236e_eb94_472b_a1a3_4fb4a72f5174.slice/crio-8079951aa064303479f53ccea673d6fec4cbaa0822a5252989d94587bcb94d8a WatchSource:0}: Error finding container 8079951aa064303479f53ccea673d6fec4cbaa0822a5252989d94587bcb94d8a: Status 404 returned error can't find the container with id 8079951aa064303479f53ccea673d6fec4cbaa0822a5252989d94587bcb94d8a
Jul 27 20:54:21.063491 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:21.063449    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" event={"ID":"d8ee236e-eb94-472b-a1a3-4fb4a72f5174","Type":"ContainerStarted","Data":"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"}
Jul 27 20:54:21.063660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:21.063500    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" event={"ID":"d8ee236e-eb94-472b-a1a3-4fb4a72f5174","Type":"ContainerStarted","Data":"8079951aa064303479f53ccea673d6fec4cbaa0822a5252989d94587bcb94d8a"}
Jul 27 20:54:21.081801 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:21.081753    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" podStartSLOduration=1.081739254 podStartE2EDuration="1.081739254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:54:21.079354599 +0000 UTC m=+3027.173940091" watchObservedRunningTime="2026-07-27 20:54:21.081739254 +0000 UTC m=+3027.176324718"
Jul 27 20:54:30.517602 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:30.517566    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:30.517602 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:30.517610    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:30.530282 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:30.530251    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:54:31.130012 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:54:31.129979    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:55:22.498826 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:22.498790    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:55:32.216880 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.216841    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7247-139727817291792"]
Jul 27 20:55:32.220940 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.220921    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:32.223213 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.223160    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dndv7\""
Jul 27 20:55:32.225140 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.225117    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7247-139727817291792"]
Jul 27 20:55:32.290681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.290641    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqqk\" (UniqueName: \"kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk\") pod \"test-curl-7247-139727817291792\" (UID: \"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91\") " pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:32.392085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.392051    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqqk\" (UniqueName: \"kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk\") pod \"test-curl-7247-139727817291792\" (UID: \"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91\") " pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:32.401279 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.401250    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqqk\" (UniqueName: \"kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk\") pod \"test-curl-7247-139727817291792\" (UID: \"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91\") " pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:32.532181 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.532149    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:32.672127 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.672100    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7247-139727817291792"]
Jul 27 20:55:32.674914 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:55:32.674883    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c6473bb_c573_4d7a_ba7a_b9f445c2ee91.slice/crio-125c87885910ba2270138f271eb6de975964e55980210d3f3492363863c285ed WatchSource:0}: Error finding container 125c87885910ba2270138f271eb6de975964e55980210d3f3492363863c285ed: Status 404 returned error can't find the container with id 125c87885910ba2270138f271eb6de975964e55980210d3f3492363863c285ed
Jul 27 20:55:32.676417 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:32.676391    2531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:55:33.473264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:33.473233    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7247-139727817291792" event={"ID":"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91","Type":"ContainerStarted","Data":"65a0a0427e86ee6cf9d1891d34006c880f5cacfd2db64e2e9a2ab8067faa5f61"}
Jul 27 20:55:33.473635 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:33.473280    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7247-139727817291792" event={"ID":"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91","Type":"ContainerStarted","Data":"125c87885910ba2270138f271eb6de975964e55980210d3f3492363863c285ed"}
Jul 27 20:55:33.488657 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:33.488607    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7247-139727817291792" podStartSLOduration=0.770195377 podStartE2EDuration="1.488591169s" totalImagesPullingTime="718.395792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:33.486469802 +0000 UTC m=+3099.581055267" watchObservedRunningTime="2026-07-27 20:55:33.488591169 +0000 UTC m=+3099.583176674"
Jul 27 20:55:33.612906 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:33.612879    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7247-139727817291792_9c6473bb-c573-4d7a-ba7a-b9f445c2ee91/test-curl-7247-139727817291792/0.log"
Jul 27 20:55:34.480048 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:34.480016    2531 generic.go:356] "Generic (PLEG): container finished" podID="9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" containerID="65a0a0427e86ee6cf9d1891d34006c880f5cacfd2db64e2e9a2ab8067faa5f61" exitCode=0
Jul 27 20:55:34.480442 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:34.480092    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7247-139727817291792" event={"ID":"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91","Type":"ContainerDied","Data":"65a0a0427e86ee6cf9d1891d34006c880f5cacfd2db64e2e9a2ab8067faa5f61"}
Jul 27 20:55:35.625198 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.625152    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:35.656524 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.656497    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7247-139727817291792"]
Jul 27 20:55:35.659982 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.659955    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7247-139727817291792"]
Jul 27 20:55:35.724846 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.724814    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rqqk\" (UniqueName: \"kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk\") pod \"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91\" (UID: \"9c6473bb-c573-4d7a-ba7a-b9f445c2ee91\") "
Jul 27 20:55:35.726928 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.726904    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk" (OuterVolumeSpecName: "kube-api-access-7rqqk") pod "9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" (UID: "9c6473bb-c573-4d7a-ba7a-b9f445c2ee91"). InnerVolumeSpecName "kube-api-access-7rqqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:35.781504 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.781470    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7247-139727817292800"]
Jul 27 20:55:35.781975 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.781960    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" containerName="test-curl-7247-139727817291792"
Jul 27 20:55:35.782028 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.781976    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" containerName="test-curl-7247-139727817291792"
Jul 27 20:55:35.782096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.782086    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" containerName="test-curl-7247-139727817291792"
Jul 27 20:55:35.785344 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.785327    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:35.789264 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.789236    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7247-139727817292800"]
Jul 27 20:55:35.826093 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.826067    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqqk\" (UniqueName: \"kubernetes.io/projected/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91-kube-api-access-7rqqk\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:35.927618 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:35.927580    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5gf5\" (UniqueName: \"kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5\") pod \"test-curl-7247-139727817292800\" (UID: \"946023b0-d0e2-416b-9880-7fc698d0adfd\") " pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:36.029235 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.029170    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5gf5\" (UniqueName: \"kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5\") pod \"test-curl-7247-139727817292800\" (UID: \"946023b0-d0e2-416b-9880-7fc698d0adfd\") " pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:36.037136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.037073    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5gf5\" (UniqueName: \"kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5\") pod \"test-curl-7247-139727817292800\" (UID: \"946023b0-d0e2-416b-9880-7fc698d0adfd\") " pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:36.097292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.097260    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:36.232591 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.232468    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7247-139727817292800"]
Jul 27 20:55:36.235544 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:55:36.235498    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod946023b0_d0e2_416b_9880_7fc698d0adfd.slice/crio-08c85f6d55c08d6e1cd0e16110d6ebd173efcee87daeee7160ab7a2dd36de411 WatchSource:0}: Error finding container 08c85f6d55c08d6e1cd0e16110d6ebd173efcee87daeee7160ab7a2dd36de411: Status 404 returned error can't find the container with id 08c85f6d55c08d6e1cd0e16110d6ebd173efcee87daeee7160ab7a2dd36de411
Jul 27 20:55:36.339801 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.339718    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6473bb-c573-4d7a-ba7a-b9f445c2ee91" path="/var/lib/kubelet/pods/9c6473bb-c573-4d7a-ba7a-b9f445c2ee91/volumes"
Jul 27 20:55:36.492232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.492199    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7247-139727817292800" event={"ID":"946023b0-d0e2-416b-9880-7fc698d0adfd","Type":"ContainerStarted","Data":"08c85f6d55c08d6e1cd0e16110d6ebd173efcee87daeee7160ab7a2dd36de411"}
Jul 27 20:55:36.493585 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.493560    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817291792"
Jul 27 20:55:36.493737 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:36.493571    2531 scope.go:117] "RemoveContainer" containerID="65a0a0427e86ee6cf9d1891d34006c880f5cacfd2db64e2e9a2ab8067faa5f61"
Jul 27 20:55:37.500938 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.500900    2531 generic.go:356] "Generic (PLEG): container finished" podID="946023b0-d0e2-416b-9880-7fc698d0adfd" containerID="3d459d1065ea503840111d9d95cf0d46a74140af7769a2dfd2e1f0c53545128e" exitCode=0
Jul 27 20:55:37.501386 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.500981    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7247-139727817292800" event={"ID":"946023b0-d0e2-416b-9880-7fc698d0adfd","Type":"ContainerDied","Data":"3d459d1065ea503840111d9d95cf0d46a74140af7769a2dfd2e1f0c53545128e"}
Jul 27 20:55:37.518360 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.518319    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" podUID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" containerName="main" containerID="cri-o://4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee" gracePeriod=45
Jul 27 20:55:37.793142 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.793116    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:55:37.854560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854511    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgr7p\" (UniqueName: \"kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854607    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854643    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854666    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854757 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854744    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854791    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm\") pod \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\" (UID: \"d8ee236e-eb94-472b-a1a3-4fb4a72f5174\") "
Jul 27 20:55:37.854965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854920    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:37.854965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854934    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home" (OuterVolumeSpecName: "home") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:37.854965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.854946    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:37.855190 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.855159    2531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:37.855190 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.855194    2531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:37.855394 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.855205    2531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:37.856834 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.856809    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p" (OuterVolumeSpecName: "kube-api-access-mgr7p") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "kube-api-access-mgr7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:37.856917 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.856875    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:55:37.856917 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.856875    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm" (OuterVolumeSpecName: "dshm") pod "d8ee236e-eb94-472b-a1a3-4fb4a72f5174" (UID: "d8ee236e-eb94-472b-a1a3-4fb4a72f5174"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:55:37.955789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.955751    2531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:37.955789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.955786    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgr7p\" (UniqueName: \"kubernetes.io/projected/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-kube-api-access-mgr7p\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:37.955789 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:37.955799    2531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ee236e-eb94-472b-a1a3-4fb4a72f5174-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:38.509231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.509192    2531 generic.go:356] "Generic (PLEG): container finished" podID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" containerID="4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee" exitCode=0
Jul 27 20:55:38.509703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.509237    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"
Jul 27 20:55:38.509703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.509280    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" event={"ID":"d8ee236e-eb94-472b-a1a3-4fb4a72f5174","Type":"ContainerDied","Data":"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"}
Jul 27 20:55:38.509703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.509322    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955" event={"ID":"d8ee236e-eb94-472b-a1a3-4fb4a72f5174","Type":"ContainerDied","Data":"8079951aa064303479f53ccea673d6fec4cbaa0822a5252989d94587bcb94d8a"}
Jul 27 20:55:38.509703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.509338    2531 scope.go:117] "RemoveContainer" containerID="4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"
Jul 27 20:55:38.519836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.519656    2531 scope.go:117] "RemoveContainer" containerID="4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"
Jul 27 20:55:38.519955 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:55:38.519933    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee\": container with ID starting with 4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee not found: ID does not exist" containerID="4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"
Jul 27 20:55:38.519996 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.519964    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee"} err="failed to get container status \"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee\": rpc error: code = NotFound desc = could not find container \"4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee\": container with ID starting with 4f72ab7d350bf209006f43ef2e19cee8616fb268c51ca001eb5fb342f11089ee not found: ID does not exist"
Jul 27 20:55:38.525631 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.525604    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:55:38.531030 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.531003    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-21ded0/rate-test-model-21ded0-kserve-655cf5cc55-7n955"]
Jul 27 20:55:38.666078 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.666056    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:38.686951 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.686926    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7247-139727817292800_946023b0-d0e2-416b-9880-7fc698d0adfd/test-curl-7247-139727817292800/0.log"
Jul 27 20:55:38.714692 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.714659    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7247-139727817292800"]
Jul 27 20:55:38.718405 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.718380    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7247-139727817292800"]
Jul 27 20:55:38.764128 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.764096    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5gf5\" (UniqueName: \"kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5\") pod \"946023b0-d0e2-416b-9880-7fc698d0adfd\" (UID: \"946023b0-d0e2-416b-9880-7fc698d0adfd\") "
Jul 27 20:55:38.766215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.766189    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5" (OuterVolumeSpecName: "kube-api-access-n5gf5") pod "946023b0-d0e2-416b-9880-7fc698d0adfd" (UID: "946023b0-d0e2-416b-9880-7fc698d0adfd"). InnerVolumeSpecName "kube-api-access-n5gf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:38.865000 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:38.864967    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5gf5\" (UniqueName: \"kubernetes.io/projected/946023b0-d0e2-416b-9880-7fc698d0adfd-kube-api-access-n5gf5\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:55:39.517292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:39.517262    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c85f6d55c08d6e1cd0e16110d6ebd173efcee87daeee7160ab7a2dd36de411"
Jul 27 20:55:39.517292 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:39.517293    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:39.527047 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:55:39.527019    2531 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7247-139727817292800\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="946023b0-d0e2-416b-9880-7fc698d0adfd" pod="opendatahub/test-curl-7247-139727817292800"
Jul 27 20:55:40.340058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:40.340028    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946023b0-d0e2-416b-9880-7fc698d0adfd" path="/var/lib/kubelet/pods/946023b0-d0e2-416b-9880-7fc698d0adfd/volumes"
Jul 27 20:55:40.340490 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:40.340476    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" path="/var/lib/kubelet/pods/d8ee236e-eb94-472b-a1a3-4fb4a72f5174/volumes"
Jul 27 20:55:41.951905 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.951872    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:55:41.952909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.952883    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" containerName="main"
Jul 27 20:55:41.952909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.952910    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" containerName="main"
Jul 27 20:55:41.953090 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.952944    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946023b0-d0e2-416b-9880-7fc698d0adfd" containerName="test-curl-7247-139727817292800"
Jul 27 20:55:41.953090 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.952953    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="946023b0-d0e2-416b-9880-7fc698d0adfd" containerName="test-curl-7247-139727817292800"
Jul 27 20:55:41.953090 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.953078    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ee236e-eb94-472b-a1a3-4fb4a72f5174" containerName="main"
Jul 27 20:55:41.953262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.953093    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="946023b0-d0e2-416b-9880-7fc698d0adfd" containerName="test-curl-7247-139727817292800"
Jul 27 20:55:41.956744 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.956722    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:41.959033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.959012    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-366a10-openshift-default-dockercfg-p5lkw\""
Jul 27 20:55:41.968750 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:41.968727    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:55:42.101352 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101319    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zncvc\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101352 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101362    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101393    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101432    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101460    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101478    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101526    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101568 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101563    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.101760 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.101600    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202390 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202307    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zncvc\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202390 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202345    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202390 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202375    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202398    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202433    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202451    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202481    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202506    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202660 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202551    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202898 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202773    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202898 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202854    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.202969 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202932    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.203011 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.202995    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.203255 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.203236    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.204777 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.204758    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.205069 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.205048    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.210396 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.210371    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.210612 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.210592    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zncvc\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc\") pod \"e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") " pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.269269 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.269229    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:42.421186 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.421138    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:55:42.422322 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:55:42.422293    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bed6b8_b165_4e9c_8c57_ea94eb8bfafd.slice/crio-75264a0a8b509efdfca5a0ede96cd08da014abfeea511e2e65f88575d7cc9fa5 WatchSource:0}: Error finding container 75264a0a8b509efdfca5a0ede96cd08da014abfeea511e2e65f88575d7cc9fa5: Status 404 returned error can't find the container with id 75264a0a8b509efdfca5a0ede96cd08da014abfeea511e2e65f88575d7cc9fa5
Jul 27 20:55:42.424626 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.424592    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:55:42.424709 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.424670    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:55:42.424748 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.424719    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:55:42.539198 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.538466    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerStarted","Data":"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"}
Jul 27 20:55:42.539198 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.538568    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerStarted","Data":"75264a0a8b509efdfca5a0ede96cd08da014abfeea511e2e65f88575d7cc9fa5"}
Jul 27 20:55:42.560850 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:42.560786    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podStartSLOduration=1.560765109 podStartE2EDuration="1.560765109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:42.558739763 +0000 UTC m=+3108.653325229" watchObservedRunningTime="2026-07-27 20:55:42.560765109 +0000 UTC m=+3108.655350576"
Jul 27 20:55:43.269972 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:43.269929    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:43.275099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:43.275073    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:43.545226 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:43.545126    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:43.546230 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:43.546203    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:55:59.588316 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:59.588275    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 20:55:59.588730 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:55:59.588368    2531 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 20:56:00.279476 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.279441    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:56:00.285961 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.285936    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.289577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.289552    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3511f6-openshift-default-dockercfg-hwdmd\""
Jul 27 20:56:00.294136 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.294075    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:56:00.412096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412064    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412113    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412142    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412167    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412199    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7p7p\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412303    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412343    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412405    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.412526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.412492    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513413 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513376    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513439    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513473    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513506    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513538    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7p7p\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513587    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513611 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513608    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513640    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513703    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.513929 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.513848    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.514096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.514045    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.514096 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.514061    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.514297 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.514271    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.514915 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.514889    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.516143 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.516122    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.516369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.516348    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.532577 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.532506    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.532867 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.532846    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7p7p\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p\") pod \"e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") " pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.601958 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.601918    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:00.750375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.750348    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:56:00.751951 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:56:00.751924    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec59ced2_7706_4f5e_a0dd_2f4044b62601.slice/crio-03d9f2849fdecf01dab80a590643203a08c9f9f2cb897f94fbb38249cac93e53 WatchSource:0}: Error finding container 03d9f2849fdecf01dab80a590643203a08c9f9f2cb897f94fbb38249cac93e53: Status 404 returned error can't find the container with id 03d9f2849fdecf01dab80a590643203a08c9f9f2cb897f94fbb38249cac93e53
Jul 27 20:56:00.754164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.754128    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:00.754283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.754220    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:00.754283 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:00.754269    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:01.659058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.659009    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerStarted","Data":"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"}
Jul 27 20:56:01.659556 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.659078    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerStarted","Data":"03d9f2849fdecf01dab80a590643203a08c9f9f2cb897f94fbb38249cac93e53"}
Jul 27 20:56:01.660665 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.660642    2531 generic.go:356] "Generic (PLEG): container finished" podID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerID="e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6" exitCode=0
Jul 27 20:56:01.660776 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.660715    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerDied","Data":"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"}
Jul 27 20:56:01.661098 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.661085    2531 scope.go:117] "RemoveContainer" containerID="e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"
Jul 27 20:56:01.662115 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.662087    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:01.662231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.662165    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:01.662300 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.662229    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:01.682029 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:01.681986    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podStartSLOduration=1.681972222 podStartE2EDuration="1.681972222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:01.680853883 +0000 UTC m=+3127.775439346" watchObservedRunningTime="2026-07-27 20:56:01.681972222 +0000 UTC m=+3127.776557685"
Jul 27 20:56:02.603049 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:02.603015    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:02.609165 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:02.609129    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:02.673058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:02.673020    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerStarted","Data":"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"}
Jul 27 20:56:02.673477 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:02.673409    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:02.674526 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:02.674492    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:03.269965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:03.269930    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:56:03.270351 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:03.270328    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 27 20:56:03.270446 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:03.270374    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 27 20:56:04.269751 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:04.269712    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 27 20:56:04.270217 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:04.269798    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 27 20:56:05.269764 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:05.269730    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 27 20:56:05.270351 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:05.270319    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 27 20:56:06.270410 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:06.270372    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Jul 27 20:56:06.270828 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:06.270433    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Jul 27 20:56:07.273533 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:07.273497    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:56:07.273990 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:07.273901    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:56:07.274676 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:07.274654    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:56:22.804270 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.804232    2531 generic.go:356] "Generic (PLEG): container finished" podID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerID="3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11" exitCode=0
Jul 27 20:56:22.804722 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.804309    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerDied","Data":"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"}
Jul 27 20:56:22.804854 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.804839    2531 scope.go:117] "RemoveContainer" containerID="3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"
Jul 27 20:56:22.806031 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.806004    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:22.806141 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.806090    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:22.806259 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:22.806141    2531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Jul 27 20:56:23.814403 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:23.814357    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerStarted","Data":"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"}
Jul 27 20:56:24.602578 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:24.602530    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:24.602782 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:24.602750    2531 patch_prober.go:28] interesting pod/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Jul 27 20:56:24.602840 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:24.602802    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Jul 27 20:56:25.459137 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:25.459103    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 20:56:25.602997 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:25.602951    2531 patch_prober.go:28] interesting pod/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Jul 27 20:56:25.603169 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:25.603027    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Jul 27 20:56:26.602954 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:26.602920    2531 patch_prober.go:28] interesting pod/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Jul 27 20:56:26.603447 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:26.602987    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Jul 27 20:56:27.602712 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:27.602674    2531 patch_prober.go:28] interesting pod/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Jul 27 20:56:27.602897 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:27.602731    2531 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Jul 27 20:56:28.607099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:28.607067    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:28.607514 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:28.607458    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:56:28.608197 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:56:28.608155    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:57:17.804804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:17.804773    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 20:57:24.096597 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.096560    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:24.100606 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.100566    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.103291 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.103263    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b060a0-serving-cert\""
Jul 27 20:57:24.111760 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.111735    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:24.128968 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.128935    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.129112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.128984    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7b5n\" (UniqueName: \"kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.129185 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.129118    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.230486 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.230442    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.230696 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.230535    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.230696 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.230559    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7b5n\" (UniqueName: \"kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.230696 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:57:24.230680    2531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-serving-cert: secret "maas-api-e2e-ipp-cleanup-b060a0-serving-cert" not found
Jul 27 20:57:24.230867 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:57:24.230746    2531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls podName:8048fd1f-63c8-4a95-87ff-0d2c72aaccae nodeName:}" failed. No retries permitted until 2026-07-27 20:57:24.73073003 +0000 UTC m=+3210.825315472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" (UID: "8048fd1f-63c8-4a95-87ff-0d2c72aaccae") : secret "maas-api-e2e-ipp-cleanup-b060a0-serving-cert" not found
Jul 27 20:57:24.231160 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.231140    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.239507 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.239484    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7b5n\" (UniqueName: \"kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.735909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.735872    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:24.738694 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:24.738665    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:25.016099 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:25.016007    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:25.163864 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:25.163833    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:25.165664 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:57:25.165633    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8048fd1f_63c8_4a95_87ff_0d2c72aaccae.slice/crio-15912aad9d64a9b0d6a18f085a24da017a343797bebc26a3fe0ca8da4204c451 WatchSource:0}: Error finding container 15912aad9d64a9b0d6a18f085a24da017a343797bebc26a3fe0ca8da4204c451: Status 404 returned error can't find the container with id 15912aad9d64a9b0d6a18f085a24da017a343797bebc26a3fe0ca8da4204c451
Jul 27 20:57:25.193483 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:25.193454    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" event={"ID":"8048fd1f-63c8-4a95-87ff-0d2c72aaccae","Type":"ContainerStarted","Data":"15912aad9d64a9b0d6a18f085a24da017a343797bebc26a3fe0ca8da4204c451"}
Jul 27 20:57:26.200539 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:26.200500    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" event={"ID":"8048fd1f-63c8-4a95-87ff-0d2c72aaccae","Type":"ContainerStarted","Data":"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"}
Jul 27 20:57:26.200980 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:26.200580    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:32.212843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:32.212808    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:32.231885 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:32.231820    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" podStartSLOduration=7.818747808 podStartE2EDuration="8.231798636s" totalImagesPullingTime="413.050828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:26.223080375 +0000 UTC m=+3212.317665862" watchObservedRunningTime="2026-07-27 20:57:32.231798636 +0000 UTC m=+3218.326384102"
Jul 27 20:57:42.163962 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.163927    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"]
Jul 27 20:57:42.167829 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.167809    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:42.170337 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.170312    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:57:42.175892 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.175870    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"]
Jul 27 20:57:42.325056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.325019    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m\" (UID: \"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:42.425734 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.425646    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m\" (UID: \"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:42.426436 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.426415    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m\" (UID: \"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:42.481888 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.481853    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:42.628058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:42.628029    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"]
Jul 27 20:57:42.630649 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:57:42.630613    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce3224b_9d1f_4a56_9dac_c97d7a2a0d2b.slice/crio-9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff WatchSource:0}: Error finding container 9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff: Status 404 returned error can't find the container with id 9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff
Jul 27 20:57:43.303285 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:43.303244    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m" event={"ID":"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b","Type":"ContainerStarted","Data":"787277eb28c3743555a628917fa2340dff8ff59b38b9548aecd3b28b76988a15"}
Jul 27 20:57:43.303718 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:43.303311    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m" event={"ID":"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b","Type":"ContainerStarted","Data":"9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff"}
Jul 27 20:57:43.323453 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:43.323401    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m" podStartSLOduration=1.323386881 podStartE2EDuration="1.323386881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:43.321351713 +0000 UTC m=+3229.415937178" watchObservedRunningTime="2026-07-27 20:57:43.323386881 +0000 UTC m=+3229.417972350"
Jul 27 20:57:44.311243 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:44.311205    2531 generic.go:356] "Generic (PLEG): container finished" podID="9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" containerID="787277eb28c3743555a628917fa2340dff8ff59b38b9548aecd3b28b76988a15" exitCode=0
Jul 27 20:57:44.311628 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:44.311298    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m" event={"ID":"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b","Type":"ContainerDied","Data":"787277eb28c3743555a628917fa2340dff8ff59b38b9548aecd3b28b76988a15"}
Jul 27 20:57:45.470222 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:45.470192    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:45.566108 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:45.566067    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca\") pod \"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b\" (UID: \"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b\") "
Jul 27 20:57:45.566513 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:45.566486    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" (UID: "9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:57:45.667994 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:45.667902    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:57:46.323975 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:46.323898    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m" event={"ID":"9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b","Type":"ContainerDied","Data":"9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff"}
Jul 27 20:57:46.324129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:46.323977    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d242317ee469daba831fa777e3faefa01647dea8141951951a2c64c9e0634ff"
Jul 27 20:57:46.324129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:46.323911    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"
Jul 27 20:57:53.170276 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.170235    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:57:53.170956 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.170873    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" containerName="revoke-keys"
Jul 27 20:57:53.170956 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.170889    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" containerName="revoke-keys"
Jul 27 20:57:53.171097 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.171002    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" containerName="revoke-keys"
Jul 27 20:57:53.174501 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.174463    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:57:53.184019 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.183973    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:57:53.250262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.250224    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghktw\" (UniqueName: \"kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw\") pod \"kuadrant-operator-catalog-8jdhp\" (UID: \"8eef3f6d-70b1-4d96-93fd-805217d37edc\") " pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:57:53.350943 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.350907    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghktw\" (UniqueName: \"kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw\") pod \"kuadrant-operator-catalog-8jdhp\" (UID: \"8eef3f6d-70b1-4d96-93fd-805217d37edc\") " pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:57:53.359749 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.359726    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghktw\" (UniqueName: \"kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw\") pod \"kuadrant-operator-catalog-8jdhp\" (UID: \"8eef3f6d-70b1-4d96-93fd-805217d37edc\") " pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:57:53.486681 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.486603    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:57:53.634429 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:53.634401    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:57:53.636400 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:57:53.636366    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eef3f6d_70b1_4d96_93fd_805217d37edc.slice/crio-2f78a41d633882b3253b4c85d863b51142ff7fde4858aae339ae373c83685a9c WatchSource:0}: Error finding container 2f78a41d633882b3253b4c85d863b51142ff7fde4858aae339ae373c83685a9c: Status 404 returned error can't find the container with id 2f78a41d633882b3253b4c85d863b51142ff7fde4858aae339ae373c83685a9c
Jul 27 20:57:54.379982 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:54.379935    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" event={"ID":"8eef3f6d-70b1-4d96-93fd-805217d37edc","Type":"ContainerStarted","Data":"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"}
Jul 27 20:57:54.380439 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:54.379999    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" event={"ID":"8eef3f6d-70b1-4d96-93fd-805217d37edc","Type":"ContainerStarted","Data":"2f78a41d633882b3253b4c85d863b51142ff7fde4858aae339ae373c83685a9c"}
Jul 27 20:57:54.897159 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:54.897088    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" podStartSLOduration=1.544979117 podStartE2EDuration="1.897065217s" totalImagesPullingTime="352.0861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:54.397750838 +0000 UTC m=+3240.492336303" watchObservedRunningTime="2026-07-27 20:57:54.897065217 +0000 UTC m=+3240.991650683"
Jul 27 20:57:54.898964 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:54.898938    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:54.899154 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:54.899134    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" podUID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" containerName="maas-api" containerID="cri-o://ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742" gracePeriod=30
Jul 27 20:57:55.178408 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.178379    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:55.269440 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.269403    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") pod \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") "
Jul 27 20:57:55.269636 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.269495    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7b5n\" (UniqueName: \"kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n\") pod \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") "
Jul 27 20:57:55.269636 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.269560    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca\") pod \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\" (UID: \"8048fd1f-63c8-4a95-87ff-0d2c72aaccae\") "
Jul 27 20:57:55.269858 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.269826    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8048fd1f-63c8-4a95-87ff-0d2c72aaccae" (UID: "8048fd1f-63c8-4a95-87ff-0d2c72aaccae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:57:55.269953 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.269934    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:57:55.271472 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.271450    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8048fd1f-63c8-4a95-87ff-0d2c72aaccae" (UID: "8048fd1f-63c8-4a95-87ff-0d2c72aaccae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:57:55.271610 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.271590    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n" (OuterVolumeSpecName: "kube-api-access-z7b5n") pod "8048fd1f-63c8-4a95-87ff-0d2c72aaccae" (UID: "8048fd1f-63c8-4a95-87ff-0d2c72aaccae"). InnerVolumeSpecName "kube-api-access-z7b5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:57:55.370702 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.370666    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:57:55.370702 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.370696    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7b5n\" (UniqueName: \"kubernetes.io/projected/8048fd1f-63c8-4a95-87ff-0d2c72aaccae-kube-api-access-z7b5n\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:57:55.386767 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.386732    2531 generic.go:356] "Generic (PLEG): container finished" podID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" containerID="ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742" exitCode=0
Jul 27 20:57:55.387164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.386777    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"
Jul 27 20:57:55.387164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.386826    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" event={"ID":"8048fd1f-63c8-4a95-87ff-0d2c72aaccae","Type":"ContainerDied","Data":"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"}
Jul 27 20:57:55.387164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.386875    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8" event={"ID":"8048fd1f-63c8-4a95-87ff-0d2c72aaccae","Type":"ContainerDied","Data":"15912aad9d64a9b0d6a18f085a24da017a343797bebc26a3fe0ca8da4204c451"}
Jul 27 20:57:55.387164 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.386904    2531 scope.go:117] "RemoveContainer" containerID="ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"
Jul 27 20:57:55.397070 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.397050    2531 scope.go:117] "RemoveContainer" containerID="ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"
Jul 27 20:57:55.397388 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:57:55.397363    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742\": container with ID starting with ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742 not found: ID does not exist" containerID="ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"
Jul 27 20:57:55.397468 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.397396    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742"} err="failed to get container status \"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742\": rpc error: code = NotFound desc = could not find container \"ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742\": container with ID starting with ba74dd6489f56d4cc38e0e9d9028aeed68801996675c713fbf7758564bdc7742 not found: ID does not exist"
Jul 27 20:57:55.411207 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.411126    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:55.417232 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:55.415581    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b060a0-696d5ffb46-dmwl8"]
Jul 27 20:57:56.342726 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:56.342686    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" path="/var/lib/kubelet/pods/8048fd1f-63c8-4a95-87ff-0d2c72aaccae/volumes"
Jul 27 20:57:57.244215 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:57.244162    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"]
Jul 27 20:57:57.248033 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:57.248007    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b060a0-5p52m"]
Jul 27 20:57:58.341478 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:57:58.341444    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b" path="/var/lib/kubelet/pods/9ce3224b-9d1f-4a56-9dac-c97d7a2a0d2b/volumes"
Jul 27 20:58:03.487077 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:03.487039    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:03.487482 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:03.487106    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:03.510852 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:03.510821    2531 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:04.462356 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:04.462330    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:04.510188 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:04.510152    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:58:06.453249 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:06.453186    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" podUID="8eef3f6d-70b1-4d96-93fd-805217d37edc" containerName="registry-server" containerID="cri-o://3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9" gracePeriod=2
Jul 27 20:58:06.711466 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:06.711403    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:06.796268 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:06.796228    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghktw\" (UniqueName: \"kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw\") pod \"8eef3f6d-70b1-4d96-93fd-805217d37edc\" (UID: \"8eef3f6d-70b1-4d96-93fd-805217d37edc\") "
Jul 27 20:58:06.798353 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:06.798324    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw" (OuterVolumeSpecName: "kube-api-access-ghktw") pod "8eef3f6d-70b1-4d96-93fd-805217d37edc" (UID: "8eef3f6d-70b1-4d96-93fd-805217d37edc"). InnerVolumeSpecName "kube-api-access-ghktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:58:06.897728 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:06.897695    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghktw\" (UniqueName: \"kubernetes.io/projected/8eef3f6d-70b1-4d96-93fd-805217d37edc-kube-api-access-ghktw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:58:07.460089 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.460053    2531 generic.go:356] "Generic (PLEG): container finished" podID="8eef3f6d-70b1-4d96-93fd-805217d37edc" containerID="3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9" exitCode=0
Jul 27 20:58:07.460558 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.460098    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp"
Jul 27 20:58:07.460558 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.460147    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" event={"ID":"8eef3f6d-70b1-4d96-93fd-805217d37edc","Type":"ContainerDied","Data":"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"}
Jul 27 20:58:07.460558 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.460219    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jdhp" event={"ID":"8eef3f6d-70b1-4d96-93fd-805217d37edc","Type":"ContainerDied","Data":"2f78a41d633882b3253b4c85d863b51142ff7fde4858aae339ae373c83685a9c"}
Jul 27 20:58:07.460558 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.460248    2531 scope.go:117] "RemoveContainer" containerID="3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"
Jul 27 20:58:07.471069 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.471050    2531 scope.go:117] "RemoveContainer" containerID="3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"
Jul 27 20:58:07.471334 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:58:07.471317    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9\": container with ID starting with 3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9 not found: ID does not exist" containerID="3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"
Jul 27 20:58:07.471383 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.471343    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9"} err="failed to get container status \"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9\": rpc error: code = NotFound desc = could not find container \"3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9\": container with ID starting with 3ecb308072be784372d820a2d7cd35f0bba26849e7cd47cd4fd0f1205827aab9 not found: ID does not exist"
Jul 27 20:58:07.482300 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.482278    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:58:07.486983 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.486962    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jdhp"]
Jul 27 20:58:07.814501 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.814469    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"]
Jul 27 20:58:07.814976 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.814963    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" containerName="maas-api"
Jul 27 20:58:07.815023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.814978    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" containerName="maas-api"
Jul 27 20:58:07.815023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.815006    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eef3f6d-70b1-4d96-93fd-805217d37edc" containerName="registry-server"
Jul 27 20:58:07.815023 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.815011    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eef3f6d-70b1-4d96-93fd-805217d37edc" containerName="registry-server"
Jul 27 20:58:07.815118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.815084    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="8048fd1f-63c8-4a95-87ff-0d2c72aaccae" containerName="maas-api"
Jul 27 20:58:07.815118 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.815094    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eef3f6d-70b1-4d96-93fd-805217d37edc" containerName="registry-server"
Jul 27 20:58:07.819730 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.819711    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:07.822127 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.822104    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:58:07.825658 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.825631    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"]
Jul 27 20:58:07.910517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:07.910481    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-366a10-znptj\" (UID: \"a5a0981b-22d1-41b9-846d-c3679130466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:08.011775 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.011739    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-366a10-znptj\" (UID: \"a5a0981b-22d1-41b9-846d-c3679130466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:08.012415 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.012397    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-366a10-znptj\" (UID: \"a5a0981b-22d1-41b9-846d-c3679130466e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:08.134077 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.133982    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:08.281664 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.281635    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"]
Jul 27 20:58:08.284117 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:58:08.284090    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a0981b_22d1_41b9_846d_c3679130466e.slice/crio-a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f WatchSource:0}: Error finding container a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f: Status 404 returned error can't find the container with id a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f
Jul 27 20:58:08.342188 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.342146    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eef3f6d-70b1-4d96-93fd-805217d37edc" path="/var/lib/kubelet/pods/8eef3f6d-70b1-4d96-93fd-805217d37edc/volumes"
Jul 27 20:58:08.468917 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:08.468885    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj" event={"ID":"a5a0981b-22d1-41b9-846d-c3679130466e","Type":"ContainerStarted","Data":"a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f"}
Jul 27 20:58:09.478090 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:09.478051    2531 generic.go:356] "Generic (PLEG): container finished" podID="a5a0981b-22d1-41b9-846d-c3679130466e" containerID="03d8216d43925a636c649f1f1c6a5b2599624985e8b39985d9c573335803acfd" exitCode=0
Jul 27 20:58:09.478488 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:09.478101    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj" event={"ID":"a5a0981b-22d1-41b9-846d-c3679130466e","Type":"ContainerDied","Data":"03d8216d43925a636c649f1f1c6a5b2599624985e8b39985d9c573335803acfd"}
Jul 27 20:58:10.631440 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:10.631417    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:10.741816 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:10.741715    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca\") pod \"a5a0981b-22d1-41b9-846d-c3679130466e\" (UID: \"a5a0981b-22d1-41b9-846d-c3679130466e\") "
Jul 27 20:58:10.742111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:10.742079    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5a0981b-22d1-41b9-846d-c3679130466e" (UID: "a5a0981b-22d1-41b9-846d-c3679130466e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:58:10.742262 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:10.742200    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a0981b-22d1-41b9-846d-c3679130466e-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:58:11.490703 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:11.490677    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"
Jul 27 20:58:11.490880 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:11.490675    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj" event={"ID":"a5a0981b-22d1-41b9-846d-c3679130466e","Type":"ContainerDied","Data":"a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f"}
Jul 27 20:58:11.490880 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:11.490800    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f3c550103018c2f8fcd73431092a8760f493c76ce6af1754ad06d8b791ac2f"
Jul 27 20:58:27.903600 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:27.903552    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"]
Jul 27 20:58:27.906092 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:27.906058    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-366a10-znptj"]
Jul 27 20:58:28.340398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:28.340362    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a0981b-22d1-41b9-846d-c3679130466e" path="/var/lib/kubelet/pods/a5a0981b-22d1-41b9-846d-c3679130466e/volumes"
Jul 27 20:58:54.662836 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:54.662794    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:58:54.677904 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:58:54.677882    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 20:59:17.658254 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:17.658217    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:59:17.658706 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:17.658452    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" containerID="cri-o://24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1" gracePeriod=30
Jul 27 20:59:18.085909 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.085859    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"]
Jul 27 20:59:18.086470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.086448    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a0981b-22d1-41b9-846d-c3679130466e" containerName="revoke-keys"
Jul 27 20:59:18.086470 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.086470    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a0981b-22d1-41b9-846d-c3679130466e" containerName="revoke-keys"
Jul 27 20:59:18.086642 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.086605    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a0981b-22d1-41b9-846d-c3679130466e" containerName="revoke-keys"
Jul 27 20:59:18.089899 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.089875    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:18.092252 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.092229    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:59:18.096966 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.096934    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"]
Jul 27 20:59:18.182101 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.182059    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p\" (UID: \"d4136cad-ca10-4d37-9a6a-19c884f01b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:18.283552 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.283507    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p\" (UID: \"d4136cad-ca10-4d37-9a6a-19c884f01b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:18.284126 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.284105    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p\" (UID: \"d4136cad-ca10-4d37-9a6a-19c884f01b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:18.402724 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.402633    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:18.561374 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.561341    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"]
Jul 27 20:59:18.563500 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:59:18.563468    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4136cad_ca10_4d37_9a6a_19c884f01b45.slice/crio-bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9 WatchSource:0}: Error finding container bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9: Status 404 returned error can't find the container with id bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9
Jul 27 20:59:18.911550 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:18.911512    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p" event={"ID":"d4136cad-ca10-4d37-9a6a-19c884f01b45","Type":"ContainerStarted","Data":"bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9"}
Jul 27 20:59:19.919869 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:19.919828    2531 generic.go:356] "Generic (PLEG): container finished" podID="d4136cad-ca10-4d37-9a6a-19c884f01b45" containerID="3990dc6eecd4160101ec3f942648f8bd477ae719a370514d4db8bb7480bf37e6" exitCode=0
Jul 27 20:59:19.920298 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:19.919886    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p" event={"ID":"d4136cad-ca10-4d37-9a6a-19c884f01b45","Type":"ContainerDied","Data":"3990dc6eecd4160101ec3f942648f8bd477ae719a370514d4db8bb7480bf37e6"}
Jul 27 20:59:21.079747 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.079717    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:21.111253 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.111217    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca\") pod \"d4136cad-ca10-4d37-9a6a-19c884f01b45\" (UID: \"d4136cad-ca10-4d37-9a6a-19c884f01b45\") "
Jul 27 20:59:21.111623 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.111601    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4136cad-ca10-4d37-9a6a-19c884f01b45" (UID: "d4136cad-ca10-4d37-9a6a-19c884f01b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:59:21.212333 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.212244    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4136cad-ca10-4d37-9a6a-19c884f01b45-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:21.933116 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.933085    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"
Jul 27 20:59:21.933308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.933119    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p" event={"ID":"d4136cad-ca10-4d37-9a6a-19c884f01b45","Type":"ContainerDied","Data":"bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9"}
Jul 27 20:59:21.933308 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:21.933166    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd8b52c80db169b2282d64c5363d0120ada15073bfdc3725051ab27eb0b818e9"
Jul 27 20:59:22.276157 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.276123    2531 patch_prober.go:28] interesting pod/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:59:22.276587 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.276211    2531 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:59:22.929663 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.929594    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:59:22.940713 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.940682    2531 generic.go:356] "Generic (PLEG): container finished" podID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerID="24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1" exitCode=0
Jul 27 20:59:22.940860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.940729    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"
Jul 27 20:59:22.940860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.940770    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerDied","Data":"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"}
Jul 27 20:59:22.940860 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.940833    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm" event={"ID":"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd","Type":"ContainerDied","Data":"75264a0a8b509efdfca5a0ede96cd08da014abfeea511e2e65f88575d7cc9fa5"}
Jul 27 20:59:22.940965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.940866    2531 scope.go:117] "RemoveContainer" containerID="24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"
Jul 27 20:59:22.951614 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.951593    2531 scope.go:117] "RemoveContainer" containerID="e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"
Jul 27 20:59:22.963016 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.962994    2531 scope.go:117] "RemoveContainer" containerID="24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"
Jul 27 20:59:22.963309 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:59:22.963290    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1\": container with ID starting with 24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1 not found: ID does not exist" containerID="24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"
Jul 27 20:59:22.963398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.963318    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1"} err="failed to get container status \"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1\": rpc error: code = NotFound desc = could not find container \"24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1\": container with ID starting with 24acc31008cb4fc6d9fb67365275234a8b189993f82a8c02870e1ba06c5368c1 not found: ID does not exist"
Jul 27 20:59:22.963398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.963337    2531 scope.go:117] "RemoveContainer" containerID="e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"
Jul 27 20:59:22.963599 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:59:22.963577    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6\": container with ID starting with e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6 not found: ID does not exist" containerID="e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"
Jul 27 20:59:22.963662 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:22.963610    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6"} err="failed to get container status \"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6\": rpc error: code = NotFound desc = could not find container \"e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6\": container with ID starting with e5e73d3e6c03b7ac008b9d5eb7ed4c19a22c75ab6ac29d466821c0a661e03ff6 not found: ID does not exist"
Jul 27 20:59:23.028319 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028278    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028337    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028354    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028372    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028404    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028428    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028473    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zncvc\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028529 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028531    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.028923 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.028614    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket\") pod \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\" (UID: \"85bed6b8-b165-4e9c-8c57-ea94eb8bfafd\") "
Jul 27 20:59:23.029085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.029031    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:23.029085 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.029049    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:23.029313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.029212    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:23.029313 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.029243    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:59:23.029398 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.029337    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data" (OuterVolumeSpecName: "istio-data") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:23.030917 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.030887    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:23.030917 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.030898    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc" (OuterVolumeSpecName: "kube-api-access-zncvc") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "kube-api-access-zncvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:59:23.031107 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.031091    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:59:23.031151 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.031106    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token" (OuterVolumeSpecName: "istio-token") pod "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" (UID: "85bed6b8-b165-4e9c-8c57-ea94eb8bfafd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:59:23.129656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129620    2531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129654    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zncvc\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-kube-api-access-zncvc\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129664    2531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129656 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129673    2531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-workload-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129681    2531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-data\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129690    2531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-podinfo\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129699    2531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-credential-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129707    2531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istio-envoy\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.129947 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.129714    2531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd-istiod-ca-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:23.264412 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.264378    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:59:23.270395 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:23.270366    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-366a10-openshift-default-5c7ffcd6b4-w4dzm"]
Jul 27 20:59:24.340487 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:24.340445    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" path="/var/lib/kubelet/pods/85bed6b8-b165-4e9c-8c57-ea94eb8bfafd/volumes"
Jul 27 20:59:33.170752 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:33.170709    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"]
Jul 27 20:59:33.175469 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:33.175438    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3511f6-wjl7p"]
Jul 27 20:59:34.341371 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:34.341340    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4136cad-ca10-4d37-9a6a-19c884f01b45" path="/var/lib/kubelet/pods/d4136cad-ca10-4d37-9a6a-19c884f01b45/volumes"
Jul 27 20:59:40.117350 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.117310    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:59:40.117855 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.117606    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" containerID="cri-o://05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7" gracePeriod=30
Jul 27 20:59:40.490520 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.490412    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"]
Jul 27 20:59:40.491010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.490990    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.491111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491013    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.491111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491024    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.491111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491031    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.491111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491053    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4136cad-ca10-4d37-9a6a-19c884f01b45" containerName="revoke-keys"
Jul 27 20:59:40.491111 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491062    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4136cad-ca10-4d37-9a6a-19c884f01b45" containerName="revoke-keys"
Jul 27 20:59:40.491369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491245    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4136cad-ca10-4d37-9a6a-19c884f01b45" containerName="revoke-keys"
Jul 27 20:59:40.491369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491265    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.491369 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.491278    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bed6b8-b165-4e9c-8c57-ea94eb8bfafd" containerName="istio-proxy"
Jul 27 20:59:40.494596 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.494572    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:40.496776 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.496754    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 20:59:40.501803 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.501776    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"]
Jul 27 20:59:40.613341 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.613302    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k\" (UID: \"ea52df48-83f1-4994-abe2-cd9ba79d2c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:40.714323 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.714286    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k\" (UID: \"ea52df48-83f1-4994-abe2-cd9ba79d2c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:40.714989 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.714969    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k\" (UID: \"ea52df48-83f1-4994-abe2-cd9ba79d2c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:40.806049 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.806011    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:40.959636 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:40.959609    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"]
Jul 27 20:59:40.962689 ip-10-0-138-123 kubenswrapper[2531]: W0727 20:59:40.962655    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea52df48_83f1_4994_abe2_cd9ba79d2c74.slice/crio-ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137 WatchSource:0}: Error finding container ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137: Status 404 returned error can't find the container with id ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137
Jul 27 20:59:41.054583 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:41.054550    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k" event={"ID":"ea52df48-83f1-4994-abe2-cd9ba79d2c74","Type":"ContainerStarted","Data":"ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137"}
Jul 27 20:59:42.062271 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:42.062238    2531 generic.go:356] "Generic (PLEG): container finished" podID="ea52df48-83f1-4994-abe2-cd9ba79d2c74" containerID="586c02566631e85ed3ba305f81e263c166a9f05692bd9ff10459939f42c63113" exitCode=0
Jul 27 20:59:42.062673 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:42.062293    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k" event={"ID":"ea52df48-83f1-4994-abe2-cd9ba79d2c74","Type":"ContainerDied","Data":"586c02566631e85ed3ba305f81e263c166a9f05692bd9ff10459939f42c63113"}
Jul 27 20:59:43.217871 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.217847    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:43.345943 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.345854    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca\") pod \"ea52df48-83f1-4994-abe2-cd9ba79d2c74\" (UID: \"ea52df48-83f1-4994-abe2-cd9ba79d2c74\") "
Jul 27 20:59:43.346343 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.346309    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea52df48-83f1-4994-abe2-cd9ba79d2c74" (UID: "ea52df48-83f1-4994-abe2-cd9ba79d2c74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:59:43.447357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.447317    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea52df48-83f1-4994-abe2-cd9ba79d2c74-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:43.609995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.609887    2531 patch_prober.go:28] interesting pod/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:59:43.609995 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:43.609970    2531 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:59:44.076544 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:44.076501    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k" event={"ID":"ea52df48-83f1-4994-abe2-cd9ba79d2c74","Type":"ContainerDied","Data":"ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137"}
Jul 27 20:59:44.076721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:44.076563    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9db16287935ea42ef3ee49b4f71735049b67f884deafbd5e213304e21ea137"
Jul 27 20:59:44.076721 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:44.076512    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"
Jul 27 20:59:45.391112 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.391050    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:59:45.465774 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.465739    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.465948 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.465813    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.465948 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.465848    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7p7p\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.465947    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466016    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466058 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466041    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466093    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:45.466250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466123    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466150    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466205    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs\") pod \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\" (UID: \"ec59ced2-7706-4f5e-a0dd-2f4044b62601\") "
Jul 27 20:59:45.466250 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466220    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:45.466560 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466536    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:59:45.466622 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466606    2531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-credential-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.466678 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466627    2531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-socket\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.466804 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.466780    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:45.467129 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.467103    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:45.468375 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.468348    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 20:59:45.468482 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.468466    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:59:45.468732 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.468706    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:59:45.468965 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.468947    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p" (OuterVolumeSpecName: "kube-api-access-t7p7p") pod "ec59ced2-7706-4f5e-a0dd-2f4044b62601" (UID: "ec59ced2-7706-4f5e-a0dd-2f4044b62601"). InnerVolumeSpecName "kube-api-access-t7p7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:59:45.568056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568020    2531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-podinfo\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568050    2531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-envoy\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568060    2531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istiod-ca-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568056 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568068    2531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-data\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568076    2531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec59ced2-7706-4f5e-a0dd-2f4044b62601-workload-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568085    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7p7p\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-kube-api-access-t7p7p\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:45.568357 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:45.568093    2531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec59ced2-7706-4f5e-a0dd-2f4044b62601-istio-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:46.090840 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.090808    2531 generic.go:356] "Generic (PLEG): container finished" podID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerID="05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7" exitCode=0
Jul 27 20:59:46.091027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.090863    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"
Jul 27 20:59:46.091027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.090897    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerDied","Data":"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"}
Jul 27 20:59:46.091027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.090990    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng" event={"ID":"ec59ced2-7706-4f5e-a0dd-2f4044b62601","Type":"ContainerDied","Data":"03d9f2849fdecf01dab80a590643203a08c9f9f2cb897f94fbb38249cac93e53"}
Jul 27 20:59:46.091027 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.091013    2531 scope.go:117] "RemoveContainer" containerID="05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"
Jul 27 20:59:46.101407 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.101384    2531 scope.go:117] "RemoveContainer" containerID="3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"
Jul 27 20:59:46.109733 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.109717    2531 scope.go:117] "RemoveContainer" containerID="05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"
Jul 27 20:59:46.110003 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:59:46.109984    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7\": container with ID starting with 05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7 not found: ID does not exist" containerID="05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"
Jul 27 20:59:46.110041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.110013    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7"} err="failed to get container status \"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7\": rpc error: code = NotFound desc = could not find container \"05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7\": container with ID starting with 05eb176fbdd6a9e905508b986ccce4771d88bf598eb94cfd609448e2c9f8b6a7 not found: ID does not exist"
Jul 27 20:59:46.110041 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.110029    2531 scope.go:117] "RemoveContainer" containerID="3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"
Jul 27 20:59:46.110322 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:59:46.110300    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11\": container with ID starting with 3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11 not found: ID does not exist" containerID="3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"
Jul 27 20:59:46.110380 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.110328    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11"} err="failed to get container status \"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11\": rpc error: code = NotFound desc = could not find container \"3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11\": container with ID starting with 3b86179c8bccb393ce1664e457ba3fad98061828e77abad68f1e21ae63c33c11 not found: ID does not exist"
Jul 27 20:59:46.125809 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.125783    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:59:46.128086 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.128066    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3511f6-openshift-default-df97dd65b-l4fng"]
Jul 27 20:59:46.341788 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:46.341701    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" path="/var/lib/kubelet/pods/ec59ced2-7706-4f5e-a0dd-2f4044b62601/volumes"
Jul 27 20:59:50.769758 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:50.769725    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:59:50.770434 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:50.770036    2531 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" podUID="9f5d49ef-2898-455c-8f77-fea0451058a1" containerName="maas-api" containerID="cri-o://dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540" gracePeriod=30
Jul 27 20:59:51.051348 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.051311    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:59:51.127293 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.127252    2531 generic.go:356] "Generic (PLEG): container finished" podID="9f5d49ef-2898-455c-8f77-fea0451058a1" containerID="dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540" exitCode=0
Jul 27 20:59:51.127517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.127310    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" event={"ID":"9f5d49ef-2898-455c-8f77-fea0451058a1","Type":"ContainerDied","Data":"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"}
Jul 27 20:59:51.127517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.127270    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"
Jul 27 20:59:51.127517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.127387    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf" event={"ID":"9f5d49ef-2898-455c-8f77-fea0451058a1","Type":"ContainerDied","Data":"293a364918a0d8cfe205331d1c6a7b9c6103c4bd67a9126637734208e7037b38"}
Jul 27 20:59:51.127517 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.127423    2531 scope.go:117] "RemoveContainer" containerID="dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"
Jul 27 20:59:51.137781 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.137755    2531 scope.go:117] "RemoveContainer" containerID="dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"
Jul 27 20:59:51.138109 ip-10-0-138-123 kubenswrapper[2531]: E0727 20:59:51.138078    2531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540\": container with ID starting with dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540 not found: ID does not exist" containerID="dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"
Jul 27 20:59:51.138231 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.138119    2531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540"} err="failed to get container status \"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540\": rpc error: code = NotFound desc = could not find container \"dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540\": container with ID starting with dea7cef8440a373e34add25156f49e9a9ae8530f8849a1fca10014d53cc3f540 not found: ID does not exist"
Jul 27 20:59:51.226843 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.226799    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpwpx\" (UniqueName: \"kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx\") pod \"9f5d49ef-2898-455c-8f77-fea0451058a1\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") "
Jul 27 20:59:51.227047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.226874    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca\") pod \"9f5d49ef-2898-455c-8f77-fea0451058a1\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") "
Jul 27 20:59:51.227047 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.226950    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls\") pod \"9f5d49ef-2898-455c-8f77-fea0451058a1\" (UID: \"9f5d49ef-2898-455c-8f77-fea0451058a1\") "
Jul 27 20:59:51.227424 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.227395    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f5d49ef-2898-455c-8f77-fea0451058a1" (UID: "9f5d49ef-2898-455c-8f77-fea0451058a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:59:51.229243 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.229220    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f5d49ef-2898-455c-8f77-fea0451058a1" (UID: "9f5d49ef-2898-455c-8f77-fea0451058a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:59:51.229322 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.229301    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx" (OuterVolumeSpecName: "kube-api-access-wpwpx") pod "9f5d49ef-2898-455c-8f77-fea0451058a1" (UID: "9f5d49ef-2898-455c-8f77-fea0451058a1"). InnerVolumeSpecName "kube-api-access-wpwpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:59:51.328240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.328108    2531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpwpx\" (UniqueName: \"kubernetes.io/projected/9f5d49ef-2898-455c-8f77-fea0451058a1-kube-api-access-wpwpx\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:51.328240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.328142    2531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d49ef-2898-455c-8f77-fea0451058a1-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:51.328240 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.328157    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f5d49ef-2898-455c-8f77-fea0451058a1-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 20:59:51.453000 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.452964    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:59:51.461010 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:51.460981    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4af3f5-bdcbb6bd9-vrtjf"]
Jul 27 20:59:52.340288 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:52.340250    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5d49ef-2898-455c-8f77-fea0451058a1" path="/var/lib/kubelet/pods/9f5d49ef-2898-455c-8f77-fea0451058a1/volumes"
Jul 27 20:59:55.571082 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:55.571013    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"]
Jul 27 20:59:55.573223 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:55.573194    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4af3f5-rsh9k"]
Jul 27 20:59:56.341110 ip-10-0-138-123 kubenswrapper[2531]: I0727 20:59:56.341078    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea52df48-83f1-4994-abe2-cd9ba79d2c74" path="/var/lib/kubelet/pods/ea52df48-83f1-4994-abe2-cd9ba79d2c74/volumes"
Jul 27 21:00:00.147232 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.147188    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"]
Jul 27 21:00:00.147810 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.147784    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea52df48-83f1-4994-abe2-cd9ba79d2c74" containerName="revoke-keys"
Jul 27 21:00:00.147994 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.147968    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea52df48-83f1-4994-abe2-cd9ba79d2c74" containerName="revoke-keys"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148009    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5d49ef-2898-455c-8f77-fea0451058a1" containerName="maas-api"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148019    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5d49ef-2898-455c-8f77-fea0451058a1" containerName="maas-api"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148039    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148047    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148071    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.148083 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148079    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.148395 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148232    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea52df48-83f1-4994-abe2-cd9ba79d2c74" containerName="revoke-keys"
Jul 27 21:00:00.148395 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148246    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.148395 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148262    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5d49ef-2898-455c-8f77-fea0451058a1" containerName="maas-api"
Jul 27 21:00:00.148395 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.148272    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec59ced2-7706-4f5e-a0dd-2f4044b62601" containerName="istio-proxy"
Jul 27 21:00:00.153226 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.153197    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"]
Jul 27 21:00:00.153347 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.153299    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"
Jul 27 21:00:00.155774 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.155749    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwpct\""
Jul 27 21:00:00.157016 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.156992    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"
Jul 27 21:00:00.159530 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.159504    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"]
Jul 27 21:00:00.163078 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.163054    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"]
Jul 27 21:00:00.170708 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.170686    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"
Jul 27 21:00:00.179542 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.179520    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"
Jul 27 21:00:00.540122 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.540089    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"]
Jul 27 21:00:00.542677 ip-10-0-138-123 kubenswrapper[2531]: W0727 21:00:00.542644    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e40f00_7d26_4f22_bb4e_788bbdd0dc0c.slice/crio-0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00 WatchSource:0}: Error finding container 0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00: Status 404 returned error can't find the container with id 0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00
Jul 27 21:00:00.567303 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:00.567274    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"]
Jul 27 21:00:00.571323 ip-10-0-138-123 kubenswrapper[2531]: W0727 21:00:00.571294    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d58a92e_99be_4e28_a72a_e684a4005d7f.slice/crio-6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171 WatchSource:0}: Error finding container 6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171: Status 404 returned error can't find the container with id 6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171
Jul 27 21:00:01.191702 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.191611    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l" event={"ID":"9d58a92e-99be-4e28-a72a-e684a4005d7f","Type":"ContainerStarted","Data":"ebd6d6272924164d8b5b86da982797f04f479b15f398d30d93c702b8931ab929"}
Jul 27 21:00:01.191702 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.191680    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l" event={"ID":"9d58a92e-99be-4e28-a72a-e684a4005d7f","Type":"ContainerStarted","Data":"6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171"}
Jul 27 21:00:01.193214 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.193166    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb" event={"ID":"83e40f00-7d26-4f22-bb4e-788bbdd0dc0c","Type":"ContainerStarted","Data":"fa5066f6897d4f981ec446c083e6a9bdbc8590e2a2f747362c3368b4220e4ded"}
Jul 27 21:00:01.193268 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.193242    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb" event={"ID":"83e40f00-7d26-4f22-bb4e-788bbdd0dc0c","Type":"ContainerStarted","Data":"0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00"}
Jul 27 21:00:01.218504 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.218454    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l" podStartSLOduration=1.218438629 podStartE2EDuration="1.218438629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:01.215804739 +0000 UTC m=+3367.310390202" watchObservedRunningTime="2026-07-27 21:00:01.218438629 +0000 UTC m=+3367.313024092"
Jul 27 21:00:01.239021 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:01.238954    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb" podStartSLOduration=1.238932894 podStartE2EDuration="1.238932894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:01.238582958 +0000 UTC m=+3367.333168424" watchObservedRunningTime="2026-07-27 21:00:01.238932894 +0000 UTC m=+3367.333518362"
Jul 27 21:00:02.200657 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:02.200612    2531 generic.go:356] "Generic (PLEG): container finished" podID="9d58a92e-99be-4e28-a72a-e684a4005d7f" containerID="ebd6d6272924164d8b5b86da982797f04f479b15f398d30d93c702b8931ab929" exitCode=0
Jul 27 21:00:02.201106 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:02.200702    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l" event={"ID":"9d58a92e-99be-4e28-a72a-e684a4005d7f","Type":"ContainerDied","Data":"ebd6d6272924164d8b5b86da982797f04f479b15f398d30d93c702b8931ab929"}
Jul 27 21:00:02.202413 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:02.202387    2531 generic.go:356] "Generic (PLEG): container finished" podID="83e40f00-7d26-4f22-bb4e-788bbdd0dc0c" containerID="fa5066f6897d4f981ec446c083e6a9bdbc8590e2a2f747362c3368b4220e4ded" exitCode=0
Jul 27 21:00:02.202539 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:02.202459    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb" event={"ID":"83e40f00-7d26-4f22-bb4e-788bbdd0dc0c","Type":"ContainerDied","Data":"fa5066f6897d4f981ec446c083e6a9bdbc8590e2a2f747362c3368b4220e4ded"}
Jul 27 21:00:03.289026 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.288989    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"]
Jul 27 21:00:03.294231 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.294154    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:03.305741 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.305667    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"]
Jul 27 21:00:03.340347 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.340306    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw\" (UID: \"98a47a1d-87cf-42d7-9fc3-83c910c7cf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:03.369034 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.369002    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"
Jul 27 21:00:03.399100 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.399073    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"
Jul 27 21:00:03.441453 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.441410    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw\" (UID: \"98a47a1d-87cf-42d7-9fc3-83c910c7cf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:03.442057 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.442040    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw\" (UID: \"98a47a1d-87cf-42d7-9fc3-83c910c7cf45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:03.612855 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.612815    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:03.748244 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:03.748213    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"]
Jul 27 21:00:03.749847 ip-10-0-138-123 kubenswrapper[2531]: W0727 21:00:03.749814    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a47a1d_87cf_42d7_9fc3_83c910c7cf45.slice/crio-4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181 WatchSource:0}: Error finding container 4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181: Status 404 returned error can't find the container with id 4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181
Jul 27 21:00:04.215763 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.215667    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb"
Jul 27 21:00:04.215763 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.215721    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753100-27cmb" event={"ID":"83e40f00-7d26-4f22-bb4e-788bbdd0dc0c","Type":"ContainerDied","Data":"0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00"}
Jul 27 21:00:04.215975 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.215770    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b7e943d0e841771bbde04c2888e7af816fb4357ec067694e63d66006de4df00"
Jul 27 21:00:04.217033 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.217009    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"
Jul 27 21:00:04.217208 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.217012    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l" event={"ID":"9d58a92e-99be-4e28-a72a-e684a4005d7f","Type":"ContainerDied","Data":"6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171"}
Jul 27 21:00:04.217208 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.217123    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b5d0765db4d2a1a09f64323e5295e28b5c87a1a2a0ebdae2a8a1e5a1be23171"
Jul 27 21:00:04.218661 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.218632    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw" event={"ID":"98a47a1d-87cf-42d7-9fc3-83c910c7cf45","Type":"ContainerStarted","Data":"57c09b23467ca2ac3714f4a68b64a4e001475ea2890fe28adff78b68527a543d"}
Jul 27 21:00:04.218779 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.218682    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw" event={"ID":"98a47a1d-87cf-42d7-9fc3-83c910c7cf45","Type":"ContainerStarted","Data":"4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181"}
Jul 27 21:00:04.233520 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:04.233476    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw" podStartSLOduration=1.233456648 podStartE2EDuration="1.233456648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:04.232939442 +0000 UTC m=+3370.327524918" watchObservedRunningTime="2026-07-27 21:00:04.233456648 +0000 UTC m=+3370.328042114"
Jul 27 21:00:05.227134 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:05.227099    2531 generic.go:356] "Generic (PLEG): container finished" podID="98a47a1d-87cf-42d7-9fc3-83c910c7cf45" containerID="57c09b23467ca2ac3714f4a68b64a4e001475ea2890fe28adff78b68527a543d" exitCode=0
Jul 27 21:00:05.227613 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:05.227221    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw" event={"ID":"98a47a1d-87cf-42d7-9fc3-83c910c7cf45","Type":"ContainerDied","Data":"57c09b23467ca2ac3714f4a68b64a4e001475ea2890fe28adff78b68527a543d"}
Jul 27 21:00:06.384347 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:06.384318    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:06.471409 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:06.471373    2531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca\") pod \"98a47a1d-87cf-42d7-9fc3-83c910c7cf45\" (UID: \"98a47a1d-87cf-42d7-9fc3-83c910c7cf45\") "
Jul 27 21:00:06.471707 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:06.471683    2531 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98a47a1d-87cf-42d7-9fc3-83c910c7cf45" (UID: "98a47a1d-87cf-42d7-9fc3-83c910c7cf45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:00:06.572686 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:06.572649    2531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a47a1d-87cf-42d7-9fc3-83c910c7cf45-maas-api-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 27 21:00:07.243796 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:07.243759    2531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"
Jul 27 21:00:07.243796 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:07.243780    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw" event={"ID":"98a47a1d-87cf-42d7-9fc3-83c910c7cf45","Type":"ContainerDied","Data":"4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181"}
Jul 27 21:00:07.244053 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:07.243827    2531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aed87e6c20598bb21952b26abfb7d447991fe3d5f6cb0d27e723a6161f43181"
Jul 27 21:00:14.003756 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:14.003720    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"]
Jul 27 21:00:14.006041 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:14.006012    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-21ded0-29753100-5xf9l"]
Jul 27 21:00:14.341942 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:14.341907    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d58a92e-99be-4e28-a72a-e684a4005d7f" path="/var/lib/kubelet/pods/9d58a92e-99be-4e28-a72a-e684a4005d7f/volumes"
Jul 27 21:00:18.348988 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:18.348949    2531 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"]
Jul 27 21:00:18.351605 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:18.351580    2531 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-21ded0-gpthw"]
Jul 27 21:00:20.338907 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:20.338874    2531 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a47a1d-87cf-42d7-9fc3-83c910c7cf45" path="/var/lib/kubelet/pods/98a47a1d-87cf-42d7-9fc3-83c910c7cf45/volumes"
Jul 27 21:00:31.226753 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.226719    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pdjld_8520ebf0-6848-4f90-8ba2-a84ae16136a1/manager/1.log"
Jul 27 21:00:31.353665 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.353633    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cmnmp_7b7e60e9-c77f-4945-a32a-206f7604151f/manager/0.log"
Jul 27 21:00:31.477521 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.477446    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c94868495-fln88_b0902d67-74fa-4f5f-a44a-aaaae91e33d9/manager/0.log"
Jul 27 21:00:31.598981 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.598950    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-898fg_c89184fc-f8df-4030-8bdf-02e0f4f28cd9/server/0.log"
Jul 27 21:00:31.720644 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.720614    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-76brs_d8262546-27c2-46a9-b3c4-f7ee53bc291a/manager/0.log"
Jul 27 21:00:31.977945 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.977919    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67944f57b-schbl_ad0e43ef-8070-4031-a84e-b7d94145a5b7/copy-manifests/0.log"
Jul 27 21:00:31.990642 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:31.990610    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67944f57b-schbl_ad0e43ef-8070-4031-a84e-b7d94145a5b7/manager/0.log"
Jul 27 21:00:32.522932 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:32.522903    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-848d79ddd8-46d9r_769523bf-f746-4e90-a6e7-a92cb240246b/maas-api/0.log"
Jul 27 21:00:32.646189 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:32.646136    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753070-2j6wt_c37340c9-9376-4a8f-8de9-8a33238d1ab7/cleanup/0.log"
Jul 27 21:00:32.770058 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:32.770030    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753085-v4q6q_2914286b-7b50-4192-aaed-1e7e678b96f3/cleanup/0.log"
Jul 27 21:00:32.888009 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:32.887936    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753100-27cmb_83e40f00-7d26-4f22-bb4e-788bbdd0dc0c/cleanup/0.log"
Jul 27 21:00:34.416131 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:34.416104    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qwdh2_f79b6153-c516-4bcb-b21f-9829d15f22d7/manager/0.log"
Jul 27 21:00:34.525715 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:34.525687    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-f9j5x_330952c1-0596-4b64-9903-42f9756a8d5e/manager/0.log"
Jul 27 21:00:34.753746 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:34.753671    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-m5wl9_5ee86923-d8ae-4a76-9fcc-a6eb98e2423d/registry-server/0.log"
Jul 27 21:00:34.876836 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:34.876804    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8wlh8_b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c/manager/0.log"
Jul 27 21:00:35.111090 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:35.111055    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6sn2t_2b70cfd5-2380-4b0c-9ec0-417a4a301732/manager/0.log"
Jul 27 21:00:35.468496 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:35.468413    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66_b55751b8-797d-4d31-a26e-fd3c64a2be52/istio-proxy/0.log"
Jul 27 21:00:35.705818 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:35.705787    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c9f4f89b5-mblfj_cfc09cef-e6af-4195-bf0d-9ba73cc92b71/kube-auth-proxy/0.log"
Jul 27 21:00:35.943545 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:35.943513    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/1.log"
Jul 27 21:00:36.056945 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:36.056907    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hwgww_32f3847f-0bdb-45ea-b166-2e3acd32619e/payload-pre-processing/0.log"
Jul 27 21:00:36.171610 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:36.171580    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 21:00:36.866735 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:36.866704    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4hx7r_1eab6677-9803-4865-a8cb-0ab2be2633dc/main/0.log"
Jul 27 21:00:37.099712 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:37.099682    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qzfsm_d61208ed-71a2-4af4-a0c5-d57efbc5097f/main/0.log"
Jul 27 21:00:45.822726 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:45.822696    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dflm4_fae3a580-5781-48b4-a8bb-8880bdcf81da/global-pull-secret-syncer/0.log"
Jul 27 21:00:45.950618 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:45.950585    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rkvwd_ad281922-6789-47b3-9d99-8920f6816fef/konnectivity-agent/0.log"
Jul 27 21:00:46.063150 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:46.063116    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-123.ec2.internal_ec91069f1c74047cfed254ac36b1a990/haproxy/0.log"
Jul 27 21:00:51.463558 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:51.463525    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qwdh2_f79b6153-c516-4bcb-b21f-9829d15f22d7/manager/0.log"
Jul 27 21:00:51.489804 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:51.489762    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-f9j5x_330952c1-0596-4b64-9903-42f9756a8d5e/manager/0.log"
Jul 27 21:00:51.566423 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:51.566337    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-m5wl9_5ee86923-d8ae-4a76-9fcc-a6eb98e2423d/registry-server/0.log"
Jul 27 21:00:51.710030 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:51.709994    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8wlh8_b6cdf9c3-8736-4d3b-a5cb-d356d34e2e7c/manager/0.log"
Jul 27 21:00:51.805588 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:51.805550    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6sn2t_2b70cfd5-2380-4b0c-9ec0-417a4a301732/manager/0.log"
Jul 27 21:00:53.504383 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.504356    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/alertmanager/0.log"
Jul 27 21:00:53.527631 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.527606    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/config-reloader/0.log"
Jul 27 21:00:53.550286 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.550218    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/kube-rbac-proxy-web/0.log"
Jul 27 21:00:53.574453 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.574408    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/kube-rbac-proxy/0.log"
Jul 27 21:00:53.595782 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.595757    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/kube-rbac-proxy-metric/0.log"
Jul 27 21:00:53.634999 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.634969    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/prom-label-proxy/0.log"
Jul 27 21:00:53.699121 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.699097    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c95292d-8c1a-4919-923c-9f84a18aa82f/init-config-reloader/0.log"
Jul 27 21:00:53.807683 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.807611    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-pkxt5_ff75a7d2-5329-46e6-86bd-2a9a017a1d73/cluster-monitoring-operator/0.log"
Jul 27 21:00:53.858988 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.858946    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mnmcb_84ea07f1-3acf-46c3-8448-8ca6ae2525f9/kube-state-metrics/0.log"
Jul 27 21:00:53.898625 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.898591    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mnmcb_84ea07f1-3acf-46c3-8448-8ca6ae2525f9/kube-rbac-proxy-main/0.log"
Jul 27 21:00:53.922797 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:53.922772    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mnmcb_84ea07f1-3acf-46c3-8448-8ca6ae2525f9/kube-rbac-proxy-self/0.log"
Jul 27 21:00:54.020610 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.020584    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftwjr_ae35d615-3039-4984-9e30-9f000f79fd8e/node-exporter/0.log"
Jul 27 21:00:54.041393 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.041356    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftwjr_ae35d615-3039-4984-9e30-9f000f79fd8e/kube-rbac-proxy/0.log"
Jul 27 21:00:54.062090 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.062021    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftwjr_ae35d615-3039-4984-9e30-9f000f79fd8e/init-textfile/0.log"
Jul 27 21:00:54.245853 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.245815    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m6ssv_a2f02f18-9da8-4415-9d4b-bda5b777f7d6/kube-rbac-proxy-main/0.log"
Jul 27 21:00:54.270277 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.270248    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m6ssv_a2f02f18-9da8-4415-9d4b-bda5b777f7d6/kube-rbac-proxy-self/0.log"
Jul 27 21:00:54.292299 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.292274    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m6ssv_a2f02f18-9da8-4415-9d4b-bda5b777f7d6/openshift-state-metrics/0.log"
Jul 27 21:00:54.348940 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.348919    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/prometheus/0.log"
Jul 27 21:00:54.373012 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.372985    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/config-reloader/0.log"
Jul 27 21:00:54.395245 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.395221    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/thanos-sidecar/0.log"
Jul 27 21:00:54.417250 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.417222    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/kube-rbac-proxy-web/0.log"
Jul 27 21:00:54.438340 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.438318    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/kube-rbac-proxy/0.log"
Jul 27 21:00:54.459729 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.459707    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/kube-rbac-proxy-thanos/0.log"
Jul 27 21:00:54.481394 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.481370    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd36787c-1664-4eea-a5bd-a2a7715c7cde/init-config-reloader/0.log"
Jul 27 21:00:54.667585 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.667517    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/thanos-query/0.log"
Jul 27 21:00:54.704234 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.704195    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/kube-rbac-proxy-web/0.log"
Jul 27 21:00:54.726790 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.726768    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/kube-rbac-proxy/0.log"
Jul 27 21:00:54.748609 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.748584    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/prom-label-proxy/0.log"
Jul 27 21:00:54.770328 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.770303    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/kube-rbac-proxy-rules/0.log"
Jul 27 21:00:54.791220 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:54.791169    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-79cf5d867-h8w7r_c192b7bd-ae2b-4523-ae0e-d149444bd64a/kube-rbac-proxy-metrics/0.log"
Jul 27 21:00:56.140827 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.140787    2531 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"]
Jul 27 21:00:56.141424 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141404    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a47a1d-87cf-42d7-9fc3-83c910c7cf45" containerName="revoke-keys"
Jul 27 21:00:56.141472 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141425    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a47a1d-87cf-42d7-9fc3-83c910c7cf45" containerName="revoke-keys"
Jul 27 21:00:56.141472 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141451    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d58a92e-99be-4e28-a72a-e684a4005d7f" containerName="cleanup"
Jul 27 21:00:56.141472 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141457    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d58a92e-99be-4e28-a72a-e684a4005d7f" containerName="cleanup"
Jul 27 21:00:56.141569 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141474    2531 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e40f00-7d26-4f22-bb4e-788bbdd0dc0c" containerName="cleanup"
Jul 27 21:00:56.141569 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141480    2531 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e40f00-7d26-4f22-bb4e-788bbdd0dc0c" containerName="cleanup"
Jul 27 21:00:56.141569 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141544    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a47a1d-87cf-42d7-9fc3-83c910c7cf45" containerName="revoke-keys"
Jul 27 21:00:56.141569 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141555    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d58a92e-99be-4e28-a72a-e684a4005d7f" containerName="cleanup"
Jul 27 21:00:56.141569 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.141564    2531 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e40f00-7d26-4f22-bb4e-788bbdd0dc0c" containerName="cleanup"
Jul 27 21:00:56.146069 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.146053    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.148433 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.148414    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpxkn\"/\"kube-root-ca.crt\""
Jul 27 21:00:56.148549 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.148443    2531 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jpxkn\"/\"default-dockercfg-z468q\""
Jul 27 21:00:56.149230 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.149210    2531 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpxkn\"/\"openshift-service-ca.crt\""
Jul 27 21:00:56.154946 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.154922    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"]
Jul 27 21:00:56.279291 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.279255    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-lib-modules\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.279494 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.279321    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-podres\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.279494 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.279343    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthgd\" (UniqueName: \"kubernetes.io/projected/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-kube-api-access-nthgd\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.279494 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.279416    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-proc\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.279494 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.279480    2531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-sys\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.380927 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.380889    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-lib-modules\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381117 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.380972    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-podres\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381117 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381001    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthgd\" (UniqueName: \"kubernetes.io/projected/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-kube-api-access-nthgd\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381117 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381044    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-proc\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381117 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381087    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-lib-modules\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381377 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381094    2531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-sys\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381377 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381159    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-sys\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381377 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381144    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-proc\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.381377 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.381146    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-podres\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.389384 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.389359    2531 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthgd\" (UniqueName: \"kubernetes.io/projected/c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad-kube-api-access-nthgd\") pod \"perf-node-gather-daemonset-cj5gr\" (UID: \"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad\") " pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.456679 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.456586    2531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:56.592895 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.592865    2531 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"]
Jul 27 21:00:56.594468 ip-10-0-138-123 kubenswrapper[2531]: W0727 21:00:56.594426    2531 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc5d7c0cc_4dac_4ee2_bd2f_3c1852b6a2ad.slice/crio-3b878d81ef779e4bb8905b44c1a9ab2f735c1fbf3136a86eb3a1c7be0b9427e9 WatchSource:0}: Error finding container 3b878d81ef779e4bb8905b44c1a9ab2f735c1fbf3136a86eb3a1c7be0b9427e9: Status 404 returned error can't find the container with id 3b878d81ef779e4bb8905b44c1a9ab2f735c1fbf3136a86eb3a1c7be0b9427e9
Jul 27 21:00:56.648351 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.648305    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/1.log"
Jul 27 21:00:56.654858 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:56.654820    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-j7tjq_f676e8c6-2bdd-4286-96ec-79acb3a93e90/console-operator/2.log"
Jul 27 21:00:57.557250 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:57.557202    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr" event={"ID":"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad","Type":"ContainerStarted","Data":"bfe45e3731704593ad62dbaee1a4e4ea41c5dc25db334e4fe11ff5350d18b623"}
Jul 27 21:00:57.557680 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:57.557264    2531 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr" event={"ID":"c5d7c0cc-4dac-4ee2-bd2f-3c1852b6a2ad","Type":"ContainerStarted","Data":"3b878d81ef779e4bb8905b44c1a9ab2f735c1fbf3136a86eb3a1c7be0b9427e9"}
Jul 27 21:00:57.557680 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:57.557293    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:00:57.576739 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:57.576692    2531 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr" podStartSLOduration=1.57667725 podStartE2EDuration="1.57667725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:57.57432601 +0000 UTC m=+3423.668911476" watchObservedRunningTime="2026-07-27 21:00:57.57667725 +0000 UTC m=+3423.671262715"
Jul 27 21:00:58.845898 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:58.845870    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmzlx_26082a5d-26b7-4e73-8131-f5446e7d9441/dns/0.log"
Jul 27 21:00:58.866855 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:58.866828    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmzlx_26082a5d-26b7-4e73-8131-f5446e7d9441/kube-rbac-proxy/0.log"
Jul 27 21:00:58.917550 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:58.917519    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f27hf_3dd1f022-a6a1-48fa-acaf-d545c56e8ad6/dns-node-resolver/0.log"
Jul 27 21:00:59.469675 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:59.469645    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d459c79d7-c5g59_a50cc35a-ab6b-4924-a712-2f44b4413c9d/registry/0.log"
Jul 27 21:00:59.544290 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:00:59.544256    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qd678_343e6847-03ce-4a40-96ea-0e0524a2110e/node-ca/0.log"
Jul 27 21:01:00.500117 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.500091    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66_b55751b8-797d-4d31-a26e-fd3c64a2be52/istio-proxy/0.log"
Jul 27 21:01:00.793429 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.793398    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c9f4f89b5-mblfj_cfc09cef-e6af-4195-bf0d-9ba73cc92b71/kube-auth-proxy/0.log"
Jul 27 21:01:00.856505 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.856479    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/0.log"
Jul 27 21:01:00.881962 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.881930    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/1.log"
Jul 27 21:01:00.912304 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.912272    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hwgww_32f3847f-0bdb-45ea-b166-2e3acd32619e/payload-pre-processing/0.log"
Jul 27 21:01:00.942966 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:00.942940    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 21:01:01.824925 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:01.824897    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-krdxc_f1586ff9-673a-43ad-a10c-31e8fe6aa7b2/serve-healthcheck-canary/0.log"
Jul 27 21:01:02.330422 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:02.330394    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-f24xp_03a10cc3-eb6e-4725-bc99-b490924fa229/insights-operator/0.log"
Jul 27 21:01:02.331012 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:02.330996    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-f24xp_03a10cc3-eb6e-4725-bc99-b490924fa229/insights-operator/1.log"
Jul 27 21:01:02.354733 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:02.354712    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cpd7_c9cfab18-fb28-417f-b0e9-7eee826ec06f/kube-rbac-proxy/0.log"
Jul 27 21:01:02.377928 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:02.377907    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cpd7_c9cfab18-fb28-417f-b0e9-7eee826ec06f/exporter/0.log"
Jul 27 21:01:02.402352 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:02.402329    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7cpd7_c9cfab18-fb28-417f-b0e9-7eee826ec06f/extractor/0.log"
Jul 27 21:01:03.575908 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:03.575876    2531 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jpxkn/perf-node-gather-daemonset-cj5gr"
Jul 27 21:01:04.711138 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:04.711111    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pdjld_8520ebf0-6848-4f90-8ba2-a84ae16136a1/manager/1.log"
Jul 27 21:01:04.711486 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:04.711382    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pdjld_8520ebf0-6848-4f90-8ba2-a84ae16136a1/manager/0.log"
Jul 27 21:01:04.927201 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:04.927096    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cmnmp_7b7e60e9-c77f-4945-a32a-206f7604151f/manager/0.log"
Jul 27 21:01:05.098226 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:05.098198    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c94868495-fln88_b0902d67-74fa-4f5f-a44a-aaaae91e33d9/manager/0.log"
Jul 27 21:01:05.121700 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:05.121673    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-898fg_c89184fc-f8df-4030-8bdf-02e0f4f28cd9/server/0.log"
Jul 27 21:01:05.214232 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:05.214109    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-76brs_d8262546-27c2-46a9-b3c4-f7ee53bc291a/manager/0.log"
Jul 27 21:01:05.383255 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:05.383224    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67944f57b-schbl_ad0e43ef-8070-4031-a84e-b7d94145a5b7/manager/0.log"
Jul 27 21:01:05.405584 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:05.405554    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67944f57b-schbl_ad0e43ef-8070-4031-a84e-b7d94145a5b7/copy-manifests/0.log"
Jul 27 21:01:06.613930 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.613899    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66_b55751b8-797d-4d31-a26e-fd3c64a2be52/istio-proxy/0.log"
Jul 27 21:01:06.807188 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.807134    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-96nrq_6046cda3-84e4-4720-a577-223b46a0337a/openshift-lws-operator/0.log"
Jul 27 21:01:06.898103 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.898029    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c9f4f89b5-mblfj_cfc09cef-e6af-4195-bf0d-9ba73cc92b71/kube-auth-proxy/0.log"
Jul 27 21:01:06.931747 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.931703    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/0.log"
Jul 27 21:01:06.961800 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.961758    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/1.log"
Jul 27 21:01:06.982185 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:06.982140    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hwgww_32f3847f-0bdb-45ea-b166-2e3acd32619e/payload-pre-processing/0.log"
Jul 27 21:01:07.003615 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:07.003579    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 21:01:08.476424 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.476388    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xfv66_b55751b8-797d-4d31-a26e-fd3c64a2be52/istio-proxy/0.log"
Jul 27 21:01:08.760274 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.760203    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c9f4f89b5-mblfj_cfc09cef-e6af-4195-bf0d-9ba73cc92b71/kube-auth-proxy/0.log"
Jul 27 21:01:08.792720 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.792686    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/0.log"
Jul 27 21:01:08.823819 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.823787    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-thjbb_88a46126-6fe1-49e3-8ad3-076099d7b9d5/istio-proxy/1.log"
Jul 27 21:01:08.850506 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.850470    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hwgww_32f3847f-0bdb-45ea-b166-2e3acd32619e/payload-pre-processing/0.log"
Jul 27 21:01:08.872777 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:08.872743    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-6vxxd_da9b250e-8139-4ef7-8128-cc1c7c228ee6/payload-processing/0.log"
Jul 27 21:01:12.055890 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:12.055851    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6p96k_9a528fdd-b5cc-4f5d-a223-991969b3a83c/migrator/0.log"
Jul 27 21:01:12.079047 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:12.079017    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-6p96k_9a528fdd-b5cc-4f5d-a223-991969b3a83c/graceful-termination/0.log"
Jul 27 21:01:12.448116 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:12.448086    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv_5209a37f-3e8b-4166-a6a7-df67e9a2db38/kube-storage-version-migrator-operator/1.log"
Jul 27 21:01:12.448963 ip-10-0-138-123 kubenswrapper[2531]: I0727 21:01:12.448940    2531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-knzhv_5209a37f-3e8b-4166-a6a7-df67e9a2db38/kube-storage-version-migrator-operator/0.log"