Jul 02 20:12:40.965832 ip-10-0-139-106 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 20:12:40.965845 ip-10-0-139-106 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 20:12:40.965855 ip-10-0-139-106 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 20:12:40.966163 ip-10-0-139-106 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 20:12:51.111010 ip-10-0-139-106 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 20:12:51.111033 ip-10-0-139-106 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 39a808fccf464103bb00f3391ff2868c -- Jul 02 20:14:50.605673 ip-10-0-139-106 systemd[1]: Starting Kubernetes Kubelet... Jul 02 20:14:51.014649 ip-10-0-139-106 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.014649 ip-10-0-139-106 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 20:14:51.014649 ip-10-0-139-106 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.014649 ip-10-0-139-106 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 20:14:51.014649 ip-10-0-139-106 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.015664 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.015580 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 20:14:51.020126 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020109 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.020126 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020126 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020129 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020132 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020135 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020140 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020144 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020147 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020150 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020152 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020159 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020163 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020166 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020169 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020171 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020174 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020177 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020179 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020182 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020185 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.020191 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020187 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020190 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020193 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020196 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020199 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020202 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020205 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020207 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020210 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020213 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020215 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020217 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020220 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020223 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020225 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020229 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020231 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020234 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020237 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020239 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.020671 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020242 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020244 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020246 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020249 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020252 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020256 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020260 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020263 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020266 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020269 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020271 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020274 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020276 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020279 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020282 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020285 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020287 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020290 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020292 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020295 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.021470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020297 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020300 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020302 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020305 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020307 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020310 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020312 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020314 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020317 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020320 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020322 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020325 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020327 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020330 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020333 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020336 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020340 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020342 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020345 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.022032 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020347 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020350 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020352 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020354 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020357 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020359 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020362 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020366 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020369 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020371 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020375 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020379 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020381 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020384 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020387 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020404 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020407 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020410 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020413 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.022516 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.020416 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.023346 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023328 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.023346 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023346 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023350 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023357 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023361 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023364 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023367 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023370 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023374 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023376 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023380 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023383 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023386 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023403 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023407 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023410 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023416 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023418 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023422 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023425 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.023422 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023428 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023431 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023434 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023437 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023440 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023444 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023447 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023450 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023456 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023459 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023462 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023465 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023468 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023471 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023474 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023477 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023480 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023483 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023486 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.023896 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023489 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023492 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023497 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023500 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023857 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023861 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023864 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023866 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023869 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023874 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023877 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023879 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023882 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023887 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023890 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023894 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023897 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023900 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023903 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023906 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.024362 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023909 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023912 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023915 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023917 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023920 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023923 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023928 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023930 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023933 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023936 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023938 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023941 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023944 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023946 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023949 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023952 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023954 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023957 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023960 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023965 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.024879 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023968 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023971 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023973 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023976 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023978 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023981 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023984 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023987 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023990 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023993 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.023995 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024003 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024006 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024008 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024011 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024013 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024016 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024020 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024024 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.025364 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024027 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024658 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024663 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024666 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024669 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024672 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024675 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024680 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024684 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024688 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024693 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024697 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024700 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024702 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024705 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024708 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024710 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024713 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024716 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.025843 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024721 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024724 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024727 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024730 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024732 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024735 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024737 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024740 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024742 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024745 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024748 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024751 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024753 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024758 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024761 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024763 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024766 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024768 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024771 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024774 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.026303 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024776 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024779 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024781 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024784 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024787 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024792 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024795 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024797 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024800 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024803 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024805 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024808 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024811 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024815 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024818 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024829 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024832 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024835 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024840 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024843 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.026784 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024845 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024848 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024851 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024853 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024856 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024859 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024861 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024864 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024867 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024870 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024876 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024880 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024884 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024887 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024891 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024894 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024897 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024901 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024903 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.027264 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024906 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024909 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024914 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024917 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024919 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024922 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024925 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024927 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024930 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024933 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024936 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024938 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024941 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024943 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024946 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024951 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024954 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024956 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024959 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024961 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.027733 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024964 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.024966 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025118 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025128 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025138 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025145 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025151 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025156 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025169 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025174 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025211 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025367 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025375 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025379 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025382 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025386 2579 flags.go:64] FLAG: --cgroup-root="" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025400 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025404 2579 flags.go:64] FLAG: --client-ca-file="" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025407 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025410 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025417 2579 flags.go:64] FLAG: --cluster-domain="" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025421 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025424 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025427 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 20:14:51.028243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025430 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025436 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025439 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025443 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025446 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025449 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025452 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025455 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025458 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025461 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025466 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025469 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025472 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025475 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025478 2579 flags.go:64] FLAG: --enable-server="true" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025481 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025487 2579 flags.go:64] FLAG: --event-burst="100" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025491 2579 flags.go:64] FLAG: --event-qps="50" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025494 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025499 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025502 2579 flags.go:64] FLAG: --eviction-hard="" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025506 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025509 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025512 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025517 2579 flags.go:64] FLAG: --eviction-soft="" Jul 02 20:14:51.028861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025520 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025524 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025529 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025531 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025536 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025541 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025543 2579 flags.go:64] FLAG: --feature-gates="" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025549 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025552 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025557 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025561 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025565 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025568 2579 flags.go:64] FLAG: --help="false" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025571 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025574 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025576 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025579 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025583 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025587 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025590 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025593 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025596 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025598 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 20:14:51.029457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025601 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025605 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025607 2579 flags.go:64] FLAG: --kube-reserved="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025611 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025614 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025617 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025620 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025622 2579 flags.go:64] FLAG: --lock-file="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025625 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025628 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025631 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025637 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025640 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025642 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025645 2579 flags.go:64] FLAG: --logging-format="text" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025648 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025651 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025654 2579 flags.go:64] FLAG: --manifest-url="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025657 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025662 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025665 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025669 2579 flags.go:64] FLAG: --max-pods="110" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025672 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025675 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025678 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 20:14:51.030028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025681 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025684 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025687 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025690 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025698 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025701 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025704 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025707 2579 flags.go:64] FLAG: --pod-cidr="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025710 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025716 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025719 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025722 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025726 2579 flags.go:64] FLAG: --port="10250" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025729 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025732 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04cdf6dce6176c067" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025735 2579 flags.go:64] FLAG: --qos-reserved="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025738 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025740 2579 flags.go:64] FLAG: --register-node="true" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025743 2579 flags.go:64] FLAG: --register-with-taints="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025752 2579 flags.go:64] FLAG: --registry-burst="10" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025756 2579 flags.go:64] FLAG: --registry-qps="5" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025758 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025761 2579 flags.go:64] FLAG: --reserved-memory="" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025765 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025768 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 20:14:51.030654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025772 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025774 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025777 2579 flags.go:64] FLAG: --runonce="false" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025780 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025783 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025786 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025789 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025792 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025795 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025798 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025800 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025803 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025806 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025809 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025812 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025815 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025817 2579 flags.go:64] FLAG: --system-cgroups="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025820 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025827 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025830 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025832 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025836 2579 flags.go:64] FLAG: --tls-min-version="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025839 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025842 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025845 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 20:14:51.031250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025848 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025851 2579 flags.go:64] FLAG: --v="2" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025858 2579 flags.go:64] FLAG: --version="false" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025863 2579 flags.go:64] FLAG: --vmodule="" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025867 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.025870 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.025995 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026000 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026003 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026006 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026009 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026011 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026014 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026017 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026020 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026022 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026025 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026027 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026030 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026032 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026034 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.031913 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026037 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026039 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026042 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026045 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026050 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026052 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026055 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026057 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026060 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026063 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026066 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026068 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026071 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026074 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026077 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026080 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026083 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026086 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026089 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.032500 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026091 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026094 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026096 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026098 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026102 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026106 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026109 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026112 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026115 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026117 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026120 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026122 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026125 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026127 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026130 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026133 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026136 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026140 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026142 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026145 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.032977 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026147 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026150 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026153 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026155 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026158 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026160 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026163 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026165 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026167 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026171 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026175 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026179 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026182 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026184 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026187 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026190 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026192 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026195 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026197 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.033482 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026200 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026202 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026205 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026207 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026210 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026213 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026215 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026218 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026220 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026224 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026231 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026234 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026237 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026240 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026243 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026246 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026248 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026251 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026253 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.033937 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026256 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026259 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026261 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026264 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026267 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026269 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.026272 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.026278 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.033360 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.033378 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.033422 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.033429 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033486 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033492 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.034402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033495 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033500 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033504 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033507 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033510 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033513 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033516 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033519 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033522 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033525 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033528 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033530 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033533 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033536 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033539 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033542 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033544 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033547 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033550 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033553 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.034762 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033555 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033558 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033562 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033564 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033567 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033569 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033572 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033575 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033578 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033580 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033583 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033585 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033588 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033590 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033593 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033595 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033598 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033600 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033603 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.035235 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033605 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033608 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033610 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033613 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033616 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033618 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033621 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033624 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033626 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033629 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033631 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033635 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033637 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033640 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033642 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033645 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033648 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033650 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033653 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.035704 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033655 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033658 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033660 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033664 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033667 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033669 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033672 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033674 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033677 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033680 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033682 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033685 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033687 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033690 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033692 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033695 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033697 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033700 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033702 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033705 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.036171 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033708 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033710 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033713 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033715 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033718 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033721 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033723 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033726 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033728 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033731 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033734 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033737 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033739 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033741 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033744 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033746 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033750 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033754 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.036716 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033757 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.033762 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033874 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033878 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033881 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033884 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033887 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033890 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033893 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033896 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033898 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033901 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033904 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033908 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033910 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.037149 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033914 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033916 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033919 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033922 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033924 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033927 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033929 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033932 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033934 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033937 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033940 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033942 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033945 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033948 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033950 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033953 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033955 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033958 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033960 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033963 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.037529 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033965 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033968 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033970 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033972 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033975 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033977 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033980 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033983 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033986 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033988 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033990 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033993 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033996 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.033998 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034001 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034003 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034006 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034009 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034011 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034013 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.038018 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034016 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034018 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034021 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034024 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034027 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034029 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034032 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034034 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034037 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034040 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034042 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034045 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034048 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034050 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034053 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034055 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034058 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034061 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034063 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.038502 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034065 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034068 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034070 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034073 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034075 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034078 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034080 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034083 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034085 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034087 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034090 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034092 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034095 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034097 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034100 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034102 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034105 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034110 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034114 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.038988 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034117 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034120 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034122 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034124 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034127 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034129 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034132 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:51.034134 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.034139 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.034249 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.036281 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.037963 2579 server.go:1019] "Starting client certificate rotation" Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.038063 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:14:51.039459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.038130 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:14:51.040611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.040594 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:14:51.045449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.045421 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:14:51.054744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.054725 2579 log.go:25] "Validated CRI v1 runtime API" Jul 02 20:14:51.057380 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.057365 2579 log.go:25] "Validated CRI v1 image API" Jul 02 20:14:51.058809 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.058795 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 20:14:51.061265 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.061243 2579 fs.go:135] Filesystem UUIDs: map[0ed039ae-9a53-4c97-ae00-cd5783c6b0ba:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e7dcd315-6460-444e-8300-c2345e1dc7f9:/dev/nvme0n1p3] Jul 02 20:14:51.061342 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.061263 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 20:14:51.066074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.066050 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:14:51.068556 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.068450 2579 manager.go:216] Machine: {Timestamp:2026-07-02 20:14:51.066951674 +0000 UTC m=+0.343782829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095214 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ae5727385000fc6a5f539d260fdd SystemUUID:ec28ae57-2738-5000-fc6a-5f539d260fdd BootID:39a808fc-cf46-4103-bb00-f3391ff2868c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:1f:02:70:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:1f:02:70:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:7b:66:6b:44:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 20:14:51.068556 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.068550 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 20:14:51.068679 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.068633 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 20:14:51.071843 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.071815 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 20:14:51.071979 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.071846 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 20:14:51.072657 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.072646 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 20:14:51.072689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.072659 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 20:14:51.072689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.072667 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:14:51.073310 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.073299 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:14:51.074677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.074666 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 20:14:51.078599 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078587 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:14:51.078725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078715 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 20:14:51.078768 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078746 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 02 20:14:51.078768 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078764 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 20:14:51.078835 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078778 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 20:14:51.078835 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078787 2579 kubelet.go:398] "Adding apiserver pod source" Jul 02 20:14:51.078835 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.078800 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 20:14:51.079163 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.079150 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:14:51.079206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.079168 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:14:51.081298 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.081279 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 20:14:51.083001 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.082987 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 20:14:51.083049 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.083018 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 20:14:51.084101 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084089 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084105 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084111 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084117 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084125 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084130 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 20:14:51.084139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084139 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 20:14:51.084291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084148 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 20:14:51.084291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084154 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 20:14:51.084291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084160 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 20:14:51.084291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084173 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 20:14:51.084291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.084179 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 20:14:51.085595 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.085584 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 20:14:51.085595 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.085594 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 20:14:51.088086 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.088065 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kh6s9" Jul 02 20:14:51.088929 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.088916 2579 server.go:1268] "Started kubelet" Jul 02 20:14:51.089066 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.089033 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 20:14:51.089138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.089047 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 20:14:51.089138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.089127 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 20:14:51.089322 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.089296 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-106.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 20:14:51.089521 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.089502 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 20:14:51.089571 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.089503 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 20:14:51.089918 ip-10-0-139-106 systemd[1]: Started Kubernetes Kubelet. Jul 02 20:14:51.090858 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.090810 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 20:14:51.092075 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.092058 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 02 20:14:51.094912 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.094889 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 20:14:51.095423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.095409 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kh6s9" Jul 02 20:14:51.096130 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.094928 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-106.ec2.internal.18be923c8399e212 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-106.ec2.internal,UID:ip-10-0-139-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-106.ec2.internal,},FirstTimestamp:2026-07-02 20:14:51.088888338 +0000 UTC m=+0.365719500,LastTimestamp:2026-07-02 20:14:51.088888338 +0000 UTC m=+0.365719500,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-106.ec2.internal,}" Jul 02 20:14:51.096771 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.096757 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 20:14:51.096828 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.096759 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 20:14:51.096933 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.096901 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.097495 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097466 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 20:14:51.097495 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097489 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 20:14:51.097495 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097459 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 20:14:51.097689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097676 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 02 20:14:51.097689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097688 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 02 20:14:51.097981 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097745 2579 factory.go:55] Registering systemd factory Jul 02 20:14:51.097981 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.097764 2579 factory.go:223] Registration of the systemd container factory successfully Jul 02 20:14:51.098081 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098065 2579 factory.go:153] Registering CRI-O factory Jul 02 20:14:51.098128 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098085 2579 factory.go:223] Registration of the crio container factory successfully Jul 02 20:14:51.098128 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098125 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 20:14:51.098217 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098143 2579 factory.go:103] Registering Raw factory Jul 02 20:14:51.098217 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098155 2579 manager.go:1196] Started watching for new ooms in manager Jul 02 20:14:51.098771 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.098747 2579 manager.go:349] Starting recovery of all containers Jul 02 20:14:51.102139 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.102103 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 20:14:51.102285 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.102256 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 20:14:51.112271 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.112153 2579 manager.go:354] Recovery completed Jul 02 20:14:51.116475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.116461 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.118470 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.118451 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.118522 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.118486 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.118522 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.118497 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.119091 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119078 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 20:14:51.119091 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119091 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 20:14:51.119194 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119101 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.119194 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119108 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:14:51.119760 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119746 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.119816 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119772 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.119816 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.119782 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.121196 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.121185 2579 policy_none.go:49] "None policy: Start" Jul 02 20:14:51.121257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.121200 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 20:14:51.121257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.121210 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 20:14:51.123154 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.123137 2579 policy_none.go:47] "Start" Jul 02 20:14:51.156716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.156699 2579 manager.go:337] "Starting Device Plugin manager" Jul 02 20:14:51.156807 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.156734 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 20:14:51.156807 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.156743 2579 server.go:87] "Starting device plugin registration server" Jul 02 20:14:51.157077 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.157064 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 20:14:51.157126 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.157080 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 20:14:51.157257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.157235 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 20:14:51.157353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.157328 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 20:14:51.157353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.157339 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 20:14:51.157890 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.157866 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 20:14:51.157969 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.157924 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.257961 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.257922 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.260133 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.259114 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.260133 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.259153 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.260133 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.259167 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.260133 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.259204 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.260638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.260610 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 20:14:51.261993 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.261976 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 20:14:51.262094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.262020 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 20:14:51.262094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.262045 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 20:14:51.262197 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.262101 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 20:14:51.264103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.264085 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:51.268725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.268674 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.268725 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.268696 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-106.ec2.internal\": node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.319210 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.319178 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.362720 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.362684 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal"] Jul 02 20:14:51.362800 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.362788 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.364811 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.364795 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.364877 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.364838 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.364877 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.364850 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.367146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367133 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.367305 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.367356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367328 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.367916 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367899 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.368006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367927 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.368006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367935 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.368006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367941 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.368006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367951 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.368006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.367963 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.370347 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.370330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.370421 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.370364 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.371295 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.371279 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.371365 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.371312 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.371365 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.371332 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.397205 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.397173 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-106.ec2.internal\" not found" node="ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.399222 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.399196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.399351 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.399226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.399695 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.399677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ef5ddb8f44f9117f4e576e103d305f9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-106.ec2.internal\" (UID: \"6ef5ddb8f44f9117f4e576e103d305f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.401720 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.401701 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-106.ec2.internal\" not found" node="ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.419437 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.419413 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.500504 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.500592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.500592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ef5ddb8f44f9117f4e576e103d305f9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-106.ec2.internal\" (UID: \"6ef5ddb8f44f9117f4e576e103d305f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.500592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ef5ddb8f44f9117f4e576e103d305f9-config\") pod \"kube-apiserver-proxy-ip-10-0-139-106.ec2.internal\" (UID: \"6ef5ddb8f44f9117f4e576e103d305f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.500592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.500705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.500573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9414945f31a6ef46e82da9cecb6478da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal\" (UID: \"9414945f31a6ef46e82da9cecb6478da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.520140 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.520091 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.620581 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.620537 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.700935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.700906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.704597 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:51.704576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:51.721552 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.721521 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.822171 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.822059 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:51.922674 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:51.922621 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:53.094597 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:53.094549 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-106.ec2.internal\" not found" Jul 02 20:14:53.108737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.108710 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 20:14:53.108898 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.108880 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 20:14:53.173558 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.173475 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 20:09:52 +0000 UTC" deadline="2028-01-06 08:12:16.779993237 +0000 UTC" Jul 02 20:14:53.173558 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.173508 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13259h57m23.606488432s" Jul 02 20:14:53.176046 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.176017 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:53.269495 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.269449 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" Jul 02 20:14:53.279816 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.279791 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:14:53.281641 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.281623 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" Jul 02 20:14:53.290655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.290637 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:14:53.323878 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:53.323828 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9414945f31a6ef46e82da9cecb6478da.slice/crio-c9a5c2cfb0d57f0617858b1b82f0ddeefb64d4fd11acf156b75d3b678b156bfa WatchSource:0}: Error finding container c9a5c2cfb0d57f0617858b1b82f0ddeefb64d4fd11acf156b75d3b678b156bfa: Status 404 returned error can't find the container with id c9a5c2cfb0d57f0617858b1b82f0ddeefb64d4fd11acf156b75d3b678b156bfa Jul 02 20:14:53.327908 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.327887 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:14:53.337334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.337285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" event={"ID":"9414945f31a6ef46e82da9cecb6478da","Type":"ContainerStarted","Data":"c9a5c2cfb0d57f0617858b1b82f0ddeefb64d4fd11acf156b75d3b678b156bfa"} Jul 02 20:14:53.374092 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:53.374066 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef5ddb8f44f9117f4e576e103d305f9.slice/crio-86d095fb1a121871fbef417c82965611b87a6ab666ef7a4467186238d20cece7 WatchSource:0}: Error finding container 86d095fb1a121871fbef417c82965611b87a6ab666ef7a4467186238d20cece7: Status 404 returned error can't find the container with id 86d095fb1a121871fbef417c82965611b87a6ab666ef7a4467186238d20cece7 Jul 02 20:14:53.380442 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.380420 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:53.750992 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.750961 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:53.997480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:53.997285 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:54.151457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.151369 2579 apiserver.go:52] "Watching apiserver" Jul 02 20:14:54.161376 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.161347 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 20:14:54.162686 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.162654 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-mrg6r","kube-system/konnectivity-agent-r5s5n","kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal","openshift-multus/multus-additional-cni-plugins-s4hd7","openshift-multus/multus-wvsc6","openshift-multus/network-metrics-daemon-8dm9k","kube-system/global-pull-secret-syncer-bx5qb","openshift-cluster-node-tuning-operator/tuned-hppsm","openshift-dns/node-resolver-5nfrl","openshift-image-registry/node-ca-tlgf8","openshift-network-diagnostics/network-check-target-twfrq","openshift-network-operator/iptables-alerter-jshjl"] Jul 02 20:14:54.165401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.165375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.168511 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.167587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.168511 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.168455 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.168511 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.168475 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.168723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.168678 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 20:14:54.168723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.168692 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 20:14:54.169871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.169164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 20:14:54.169871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.169168 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t72zp\"" Jul 02 20:14:54.169871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.169699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.170053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.170034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-96pdj\"" Jul 02 20:14:54.170319 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.170298 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.170426 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.170323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.172597 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.172334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.173264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.173243 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.173447 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.173433 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-24z9b\"" Jul 02 20:14:54.174478 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.174011 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 20:14:54.174478 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.174399 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.175192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.175169 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.175530 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.175511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 20:14:54.175701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.175684 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5srvv\"" Jul 02 20:14:54.175701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.175694 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.177261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.175974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.177261 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.176047 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:14:54.178412 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.178388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.180973 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.180100 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:14:54.180973 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.180619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 20:14:54.181126 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.181206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-multus\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.181276 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-kubelet\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.181276 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkp9q\" (UniqueName: \"kubernetes.io/projected/60ae01ac-7f49-40e6-a2a7-7faabd755c21-kube-api-access-vkp9q\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.181373 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.181184 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:14:54.181373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c555238c-220f-4d0b-ae19-5f4fa5095ba1-tmp-dir\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.181373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd31db36-e500-4131-8350-76ef7845ded0-iptables-alerter-script\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.181527 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.181527 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-netns\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.181623 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181573 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rchhd\"" Jul 02 20:14:54.181896 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.181881 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 20:14:54.182294 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-hostroot\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-daemon-config\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-multus-certs\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-etc-kubernetes\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cni-binary-copy\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-socket-dir-parent\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c555238c-220f-4d0b-ae19-5f4fa5095ba1-hosts-file\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.182546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66cg\" (UniqueName: \"kubernetes.io/projected/dd31db36-e500-4131-8350-76ef7845ded0-kube-api-access-x66cg\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.182546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-k8s-cni-cncf-io\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-bin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-conf-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87pg\" (UniqueName: \"kubernetes.io/projected/c555238c-220f-4d0b-ae19-5f4fa5095ba1-kube-api-access-c87pg\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd31db36-e500-4131-8350-76ef7845ded0-host-slash\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-system-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cnibin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.182779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.182743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-os-release\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.183473 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.183454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:54.183546 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.183528 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:14:54.187343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.187325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.189352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.189299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.191204 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.191082 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 20:14:54.191295 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.191207 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 20:14:54.191295 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.191261 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pwzwz\"" Jul 02 20:14:54.192801 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.191881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.192801 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.192058 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tkbjk\"" Jul 02 20:14:54.192801 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.192302 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.192801 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.192605 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.194207 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.194188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.195874 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.195806 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 20:14:54.196165 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.196125 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.196165 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.196147 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 20:14:54.196165 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.196156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 20:14:54.196165 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.196165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2ts52\"" Jul 02 20:14:54.197519 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197454 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.197519 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 20:14:54.197519 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197516 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltlnm\"" Jul 02 20:14:54.197789 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197501 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 20:14:54.197789 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 20:14:54.198069 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.197818 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 20:14:54.200613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.200592 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-trcv6" Jul 02 20:14:54.207214 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.207176 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trcv6" Jul 02 20:14:54.270116 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.270086 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 20:14:54.283897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.283864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9c8e8e5-5210-4889-9203-4d0b052a08c4-serviceca\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.283897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.283905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.283935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-os-release\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.283963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.283987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xjz\" (UniqueName: \"kubernetes.io/projected/e2f85b4d-8523-46da-b6db-d0fba83ed62d-kube-api-access-r4xjz\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-system-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c555238c-220f-4d0b-ae19-5f4fa5095ba1-tmp-dir\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgp6\" (UniqueName: \"kubernetes.io/projected/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-kube-api-access-mkgp6\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-system-cni-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.284158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbcad745-a614-4cca-a2b8-f36c660d9251-konnectivity-ca\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-etc-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66cg\" (UniqueName: \"kubernetes.io/projected/dd31db36-e500-4131-8350-76ef7845ded0-kube-api-access-x66cg\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-env-overrides\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-run\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-socket-dir-parent\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ljm\" (UniqueName: \"kubernetes.io/projected/b9c8e8e5-5210-4889-9203-4d0b052a08c4-kube-api-access-g8ljm\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-netd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-config\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-kubelet-config\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-dbus\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-k8s-cni-cncf-io\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-device-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.284691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-lib-modules\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cnibin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-os-release\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-multus\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkp9q\" (UniqueName: \"kubernetes.io/projected/60ae01ac-7f49-40e6-a2a7-7faabd755c21-kube-api-access-vkp9q\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-slash\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-bin\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-systemd\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-etc-kubernetes\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c555238c-220f-4d0b-ae19-5f4fa5095ba1-hosts-file\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99vs\" (UniqueName: \"kubernetes.io/projected/8ee91690-c8a4-46e3-acec-d37434cd50c9-kube-api-access-w99vs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cnibin\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-ovn\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-sys\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-netns\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysconfig\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-host\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znf2n\" (UniqueName: \"kubernetes.io/projected/7c032036-fc2d-4dc2-a98c-6bc734d95c38-kube-api-access-znf2n\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-conf-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c8e8e5-5210-4889-9203-4d0b052a08c4-host\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-registration-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovn-node-metrics-cert\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-kubelet\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd31db36-e500-4131-8350-76ef7845ded0-iptables-alerter-script\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd31db36-e500-4131-8350-76ef7845ded0-host-slash\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.284988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-tuned\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-hostroot\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-daemon-config\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-multus-certs\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-log-socket\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.285889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-script-lib\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-tmp\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-kubelet\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-var-lib-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87pg\" (UniqueName: \"kubernetes.io/projected/c555238c-220f-4d0b-ae19-5f4fa5095ba1-kube-api-access-c87pg\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbcad745-a614-4cca-a2b8-f36c660d9251-agent-certs\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-systemd-units\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-var-lib-kubelet\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-sys-fs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-modprobe-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-kubernetes\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-netns\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-systemd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.286555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-conf\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frb7x\" (UniqueName: \"kubernetes.io/projected/29565124-aee4-4ff1-a5a6-f61f8bbf605d-kube-api-access-frb7x\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cni-binary-copy\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-socket-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-node-log\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-bin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-bin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-system-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.285832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c555238c-220f-4d0b-ae19-5f4fa5095ba1-tmp-dir\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286302 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-kubelet\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dd31db36-e500-4131-8350-76ef7845ded0-host-slash\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-socket-dir-parent\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-os-release\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286591 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-netns\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286616 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-cni-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-etc-kubernetes\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286628 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-var-lib-cni-multus\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.287344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cni-binary-copy\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-conf-dir\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-hostroot\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c555238c-220f-4d0b-ae19-5f4fa5095ba1-hosts-file\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-k8s-cni-cncf-io\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-cnibin\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60ae01ac-7f49-40e6-a2a7-7faabd755c21-host-run-multus-certs\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.286889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dd31db36-e500-4131-8350-76ef7845ded0-iptables-alerter-script\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.288192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.287041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60ae01ac-7f49-40e6-a2a7-7faabd755c21-multus-daemon-config\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.293805 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.293726 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 20:14:54.297429 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.297404 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66cg\" (UniqueName: \"kubernetes.io/projected/dd31db36-e500-4131-8350-76ef7845ded0-kube-api-access-x66cg\") pod \"iptables-alerter-jshjl\" (UID: \"dd31db36-e500-4131-8350-76ef7845ded0\") " pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.297564 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.297427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkp9q\" (UniqueName: \"kubernetes.io/projected/60ae01ac-7f49-40e6-a2a7-7faabd755c21-kube-api-access-vkp9q\") pod \"multus-wvsc6\" (UID: \"60ae01ac-7f49-40e6-a2a7-7faabd755c21\") " pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.297764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.297749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87pg\" (UniqueName: \"kubernetes.io/projected/c555238c-220f-4d0b-ae19-5f4fa5095ba1-kube-api-access-c87pg\") pod \"node-resolver-5nfrl\" (UID: \"c555238c-220f-4d0b-ae19-5f4fa5095ba1\") " pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.340099 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.340063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" event={"ID":"6ef5ddb8f44f9117f4e576e103d305f9","Type":"ContainerStarted","Data":"86d095fb1a121871fbef417c82965611b87a6ab666ef7a4467186238d20cece7"} Jul 02 20:14:54.385916 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.385883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.385916 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.385920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-tuned\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.385947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-log-socket\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.385968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-script-lib\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.385991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-tmp\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-kubelet\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-var-lib-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbcad745-a614-4cca-a2b8-f36c660d9251-agent-certs\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-log-socket\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-systemd-units\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386129 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-var-lib-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-systemd-units\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-var-lib-kubelet\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-sys-fs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-modprobe-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-kubernetes\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-systemd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-conf\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frb7x\" (UniqueName: \"kubernetes.io/projected/29565124-aee4-4ff1-a5a6-f61f8bbf605d-kube-api-access-frb7x\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-socket-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386399 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-modprobe-d\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.386510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-node-log\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-node-log\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9c8e8e5-5210-4889-9203-4d0b052a08c4-serviceca\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386476 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-kubernetes\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-os-release\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xjz\" (UniqueName: \"kubernetes.io/projected/e2f85b4d-8523-46da-b6db-d0fba83ed62d-kube-api-access-r4xjz\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.386559 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgp6\" (UniqueName: \"kubernetes.io/projected/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-kube-api-access-mkgp6\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-system-cni-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.386672 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.886610121 +0000 UTC m=+3.091996733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbcad745-a614-4cca-a2b8-f36c660d9251-konnectivity-ca\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386714 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-kubelet\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-etc-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.387995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-env-overrides\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-run\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ljm\" (UniqueName: \"kubernetes.io/projected/b9c8e8e5-5210-4889-9203-4d0b052a08c4-kube-api-access-g8ljm\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386868 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysctl-conf\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-netd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-var-lib-kubelet\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387071 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-script-lib\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.386927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-config\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-binary-copy\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387180 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-sys-fs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovnkube-config\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e2f85b4d-8523-46da-b6db-d0fba83ed62d-env-overrides\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.388815 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-systemd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.387999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbcad745-a614-4cca-a2b8-f36c660d9251-konnectivity-ca\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9c8e8e5-5210-4889-9203-4d0b052a08c4-serviceca\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-netd\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388091 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-socket-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-kubelet-config\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-system-cni-dir\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388075 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-etc-openvswitch\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.388152 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-dbus\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.388210 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.88818983 +0000 UTC m=+3.093576440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388230 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-os-release\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388291 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-run\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-kubelet-config\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.389608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-device-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b6a8c413-e177-4d35-95d7-43f15b643628-dbus\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-lib-modules\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-lib-modules\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388576 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-device-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-slash\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-bin\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-slash\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-cni-bin\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.388879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-systemd\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-systemd\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99vs\" (UniqueName: \"kubernetes.io/projected/8ee91690-c8a4-46e3-acec-d37434cd50c9-kube-api-access-w99vs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cnibin\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.390184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-ovn\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c032036-fc2d-4dc2-a98c-6bc734d95c38-cnibin\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-sys\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-sys\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-netns\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-run-ovn\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysconfig\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389363 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2f85b4d-8523-46da-b6db-d0fba83ed62d-host-run-netns\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-host\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znf2n\" (UniqueName: \"kubernetes.io/projected/7c032036-fc2d-4dc2-a98c-6bc734d95c38-kube-api-access-znf2n\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389418 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-sysconfig\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c8e8e5-5210-4889-9203-4d0b052a08c4-host\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-registration-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbcad745-a614-4cca-a2b8-f36c660d9251-agent-certs\") pod \"konnectivity-agent-r5s5n\" (UID: \"dbcad745-a614-4cca-a2b8-f36c660d9251\") " pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ee91690-c8a4-46e3-acec-d37434cd50c9-registration-dir\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9c8e8e5-5210-4889-9203-4d0b052a08c4-host\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovn-node-metrics-cert\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389494 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29565124-aee4-4ff1-a5a6-f61f8bbf605d-host\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.390765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.389673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-tmp\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.391999 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.391981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2f85b4d-8523-46da-b6db-d0fba83ed62d-ovn-node-metrics-cert\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.394153 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.394129 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:54.394251 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.394158 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:54.394251 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.394184 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:54.394366 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.394265 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.894245273 +0000 UTC m=+3.099631907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:54.396056 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.396030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frb7x\" (UniqueName: \"kubernetes.io/projected/29565124-aee4-4ff1-a5a6-f61f8bbf605d-kube-api-access-frb7x\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.396625 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.396605 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xjz\" (UniqueName: \"kubernetes.io/projected/e2f85b4d-8523-46da-b6db-d0fba83ed62d-kube-api-access-r4xjz\") pod \"ovnkube-node-mrg6r\" (UID: \"e2f85b4d-8523-46da-b6db-d0fba83ed62d\") " pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.398051 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.398030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgp6\" (UniqueName: \"kubernetes.io/projected/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-kube-api-access-mkgp6\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.398454 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.398429 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znf2n\" (UniqueName: \"kubernetes.io/projected/7c032036-fc2d-4dc2-a98c-6bc734d95c38-kube-api-access-znf2n\") pod \"multus-additional-cni-plugins-s4hd7\" (UID: \"7c032036-fc2d-4dc2-a98c-6bc734d95c38\") " pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.398547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.398429 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ljm\" (UniqueName: \"kubernetes.io/projected/b9c8e8e5-5210-4889-9203-4d0b052a08c4-kube-api-access-g8ljm\") pod \"node-ca-tlgf8\" (UID: \"b9c8e8e5-5210-4889-9203-4d0b052a08c4\") " pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.398971 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.398954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99vs\" (UniqueName: \"kubernetes.io/projected/8ee91690-c8a4-46e3-acec-d37434cd50c9-kube-api-access-w99vs\") pod \"aws-ebs-csi-driver-node-nsh6q\" (UID: \"8ee91690-c8a4-46e3-acec-d37434cd50c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.399052 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.398988 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/29565124-aee4-4ff1-a5a6-f61f8bbf605d-etc-tuned\") pod \"tuned-hppsm\" (UID: \"29565124-aee4-4ff1-a5a6-f61f8bbf605d\") " pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.479390 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.479296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wvsc6" Jul 02 20:14:54.495156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.495125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5nfrl" Jul 02 20:14:54.505917 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.505896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jshjl" Jul 02 20:14:54.512551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.512528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" Jul 02 20:14:54.519020 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.518999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" Jul 02 20:14:54.526677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.526656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:14:54.533344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.533324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hppsm" Jul 02 20:14:54.541074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.541052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:14:54.548137 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.548116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tlgf8" Jul 02 20:14:54.893245 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.893163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:54.893245 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.893202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:54.893455 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.893317 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.893455 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.893332 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:54.893455 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.893380 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:55.893363796 +0000 UTC m=+4.098750429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.893455 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.893413 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:55.893395888 +0000 UTC m=+4.098782498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:54.994423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:54.994387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:54.994590 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.994581 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:54.994631 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.994597 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:54.994631 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.994607 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:54.994720 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:54.994659 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:14:55.994646528 +0000 UTC m=+4.200033148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:55.211592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.211357 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:09:54 +0000 UTC" deadline="2028-03-22 02:54:38.733892232 +0000 UTC" Jul 02 20:14:55.211592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.211416 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15078h39m43.52248059s" Jul 02 20:14:55.216967 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.216751 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c032036_fc2d_4dc2_a98c_6bc734d95c38.slice/crio-ee48ead59a68445863bd87c4d98b302ecdf44ec66cade480cbd0908196b9348f WatchSource:0}: Error finding container ee48ead59a68445863bd87c4d98b302ecdf44ec66cade480cbd0908196b9348f: Status 404 returned error can't find the container with id ee48ead59a68445863bd87c4d98b302ecdf44ec66cade480cbd0908196b9348f Jul 02 20:14:55.218993 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.218931 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29565124_aee4_4ff1_a5a6_f61f8bbf605d.slice/crio-d112f23df55765b07e95b2c6e1ce38c44340f087547a7fdbeef991fd35bee8c1 WatchSource:0}: Error finding container d112f23df55765b07e95b2c6e1ce38c44340f087547a7fdbeef991fd35bee8c1: Status 404 returned error can't find the container with id d112f23df55765b07e95b2c6e1ce38c44340f087547a7fdbeef991fd35bee8c1 Jul 02 20:14:55.219768 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.219655 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ae01ac_7f49_40e6_a2a7_7faabd755c21.slice/crio-c57d4eee03789357443bc84d0ac2485b28761fd3e6b6a2122417c090ce96507f WatchSource:0}: Error finding container c57d4eee03789357443bc84d0ac2485b28761fd3e6b6a2122417c090ce96507f: Status 404 returned error can't find the container with id c57d4eee03789357443bc84d0ac2485b28761fd3e6b6a2122417c090ce96507f Jul 02 20:14:55.222287 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.222252 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc555238c_220f_4d0b_ae19_5f4fa5095ba1.slice/crio-8573403deed4d88b5cd8ccfc5e319fff2660893e98377ec171cab118acddf3ae WatchSource:0}: Error finding container 8573403deed4d88b5cd8ccfc5e319fff2660893e98377ec171cab118acddf3ae: Status 404 returned error can't find the container with id 8573403deed4d88b5cd8ccfc5e319fff2660893e98377ec171cab118acddf3ae Jul 02 20:14:55.222981 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.222960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f85b4d_8523_46da_b6db_d0fba83ed62d.slice/crio-de6b887c7739fc01dfe5a30c92cbbcaf537cad37f776eb964b70c57eab0fabf8 WatchSource:0}: Error finding container de6b887c7739fc01dfe5a30c92cbbcaf537cad37f776eb964b70c57eab0fabf8: Status 404 returned error can't find the container with id de6b887c7739fc01dfe5a30c92cbbcaf537cad37f776eb964b70c57eab0fabf8 Jul 02 20:14:55.224064 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.224046 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd31db36_e500_4131_8350_76ef7845ded0.slice/crio-74489b1ee2e663f75adfdaccdb45f8b93f7b4a8eeab380c45a66bea7dfa3a4f5 WatchSource:0}: Error finding container 74489b1ee2e663f75adfdaccdb45f8b93f7b4a8eeab380c45a66bea7dfa3a4f5: Status 404 returned error can't find the container with id 74489b1ee2e663f75adfdaccdb45f8b93f7b4a8eeab380c45a66bea7dfa3a4f5 Jul 02 20:14:55.230906 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.230883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c8e8e5_5210_4889_9203_4d0b052a08c4.slice/crio-33a1a56094790a534097bb187f7ed49763ac69d0de264421b046f5878ce6a790 WatchSource:0}: Error finding container 33a1a56094790a534097bb187f7ed49763ac69d0de264421b046f5878ce6a790: Status 404 returned error can't find the container with id 33a1a56094790a534097bb187f7ed49763ac69d0de264421b046f5878ce6a790 Jul 02 20:14:55.231981 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.231952 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee91690_c8a4_46e3_acec_d37434cd50c9.slice/crio-0fccf7ddab29358149db232b6ca53744f5c9971bca9de753536f28351bd7b63d WatchSource:0}: Error finding container 0fccf7ddab29358149db232b6ca53744f5c9971bca9de753536f28351bd7b63d: Status 404 returned error can't find the container with id 0fccf7ddab29358149db232b6ca53744f5c9971bca9de753536f28351bd7b63d Jul 02 20:14:55.232776 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:14:55.232750 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcad745_a614_4cca_a2b8_f36c660d9251.slice/crio-53a3ddefd40301875fd978921bfa289c64e3c011fe3527a8c13c9e1d9e3b52ea WatchSource:0}: Error finding container 53a3ddefd40301875fd978921bfa289c64e3c011fe3527a8c13c9e1d9e3b52ea: Status 404 returned error can't find the container with id 53a3ddefd40301875fd978921bfa289c64e3c011fe3527a8c13c9e1d9e3b52ea Jul 02 20:14:55.344808 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.344773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvsc6" event={"ID":"60ae01ac-7f49-40e6-a2a7-7faabd755c21","Type":"ContainerStarted","Data":"c57d4eee03789357443bc84d0ac2485b28761fd3e6b6a2122417c090ce96507f"} Jul 02 20:14:55.346004 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.345977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" event={"ID":"8ee91690-c8a4-46e3-acec-d37434cd50c9","Type":"ContainerStarted","Data":"0fccf7ddab29358149db232b6ca53744f5c9971bca9de753536f28351bd7b63d"} Jul 02 20:14:55.347058 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.347038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlgf8" event={"ID":"b9c8e8e5-5210-4889-9203-4d0b052a08c4","Type":"ContainerStarted","Data":"33a1a56094790a534097bb187f7ed49763ac69d0de264421b046f5878ce6a790"} Jul 02 20:14:55.348769 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.348729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jshjl" event={"ID":"dd31db36-e500-4131-8350-76ef7845ded0","Type":"ContainerStarted","Data":"74489b1ee2e663f75adfdaccdb45f8b93f7b4a8eeab380c45a66bea7dfa3a4f5"} Jul 02 20:14:55.350795 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.350751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"de6b887c7739fc01dfe5a30c92cbbcaf537cad37f776eb964b70c57eab0fabf8"} Jul 02 20:14:55.352206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.352183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hppsm" event={"ID":"29565124-aee4-4ff1-a5a6-f61f8bbf605d","Type":"ContainerStarted","Data":"d112f23df55765b07e95b2c6e1ce38c44340f087547a7fdbeef991fd35bee8c1"} Jul 02 20:14:55.354771 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.354748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerStarted","Data":"ee48ead59a68445863bd87c4d98b302ecdf44ec66cade480cbd0908196b9348f"} Jul 02 20:14:55.357297 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.357273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" event={"ID":"6ef5ddb8f44f9117f4e576e103d305f9","Type":"ContainerStarted","Data":"9fdf91a6ef9561b05d3bfefb28df0d2c3808368d7eedea060f8d160fc2c72c5a"} Jul 02 20:14:55.359983 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.359955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5s5n" event={"ID":"dbcad745-a614-4cca-a2b8-f36c660d9251","Type":"ContainerStarted","Data":"53a3ddefd40301875fd978921bfa289c64e3c011fe3527a8c13c9e1d9e3b52ea"} Jul 02 20:14:55.360991 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.360971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5nfrl" event={"ID":"c555238c-220f-4d0b-ae19-5f4fa5095ba1","Type":"ContainerStarted","Data":"8573403deed4d88b5cd8ccfc5e319fff2660893e98377ec171cab118acddf3ae"} Jul 02 20:14:55.371092 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.371042 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-106.ec2.internal" podStartSLOduration=2.371028173 podStartE2EDuration="2.371028173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:14:55.370290208 +0000 UTC m=+3.575676834" watchObservedRunningTime="2026-07-02 20:14:55.371028173 +0000 UTC m=+3.576414796" Jul 02 20:14:55.901308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.901207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:55.901308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:55.901272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:55.901532 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:55.901366 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:55.901532 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:55.901385 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:55.901532 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:55.901433 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:57.90141635 +0000 UTC m=+6.106802957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:55.901532 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:55.901450 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:57.901441219 +0000 UTC m=+6.106827832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:56.002960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.002298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:56.002960 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.002485 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:56.002960 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.002504 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:56.002960 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.002517 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:56.002960 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.002575 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:14:58.002557857 +0000 UTC m=+6.207944490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:56.212297 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.212164 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:09:54 +0000 UTC" deadline="2028-04-16 08:56:47.823743226 +0000 UTC" Jul 02 20:14:56.212297 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.212208 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15684h41m51.611540381s" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.336709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.336867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.337284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.337373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.337456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:56.337611 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:56.337529 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:14:56.378070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.377480 2579 generic.go:356] "Generic (PLEG): container finished" podID="9414945f31a6ef46e82da9cecb6478da" containerID="6dedc63ceeee5ec41fb654f471cdc40e2f401a7712f76acad13c62d5213bc5a5" exitCode=0 Jul 02 20:14:56.378070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:56.377616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" event={"ID":"9414945f31a6ef46e82da9cecb6478da","Type":"ContainerDied","Data":"6dedc63ceeee5ec41fb654f471cdc40e2f401a7712f76acad13c62d5213bc5a5"} Jul 02 20:14:57.411503 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:57.411458 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" event={"ID":"9414945f31a6ef46e82da9cecb6478da","Type":"ContainerStarted","Data":"6383d2e218b82a1f96a441a93d4a7623146e9d68c3531bb85729e9a51a1d9a18"} Jul 02 20:14:57.426853 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:57.426780 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-106.ec2.internal" podStartSLOduration=4.426759607 podStartE2EDuration="4.426759607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:14:57.42661406 +0000 UTC m=+5.632000720" watchObservedRunningTime="2026-07-02 20:14:57.426759607 +0000 UTC m=+5.632146220" Jul 02 20:14:57.921057 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:57.921020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:57.921258 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:57.921071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:57.921258 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:57.921216 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:57.921358 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:57.921278 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:01.92126119 +0000 UTC m=+10.126647801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:57.921695 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:57.921674 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:57.921800 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:57.921731 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:01.921716485 +0000 UTC m=+10.127103092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:58.022426 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:58.022383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:58.022617 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.022565 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:58.022617 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.022588 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:58.022617 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.022601 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:58.022775 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.022664 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:15:02.022643619 +0000 UTC m=+10.228030249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:58.335030 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:58.334948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:14:58.335194 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.335078 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:14:58.335452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:58.334948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:14:58.335577 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.335555 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:14:58.335877 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:14:58.335735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:14:58.335877 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:14:58.335821 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:00.334480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:00.334622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:00.334647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:00.334697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:00.334784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:00.335059 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:00.334890 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:01.955722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:01.955777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:01.955948 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:01.955998 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:09.955984656 +0000 UTC m=+18.161371264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:01.956409 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:01.956503 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:01.956461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:09.956445228 +0000 UTC m=+18.161831850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:02.056728 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:02.056673 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:02.056953 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.056854 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:02.056953 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.056879 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:02.056953 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.056890 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:02.056953 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.056946 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:15:10.056926821 +0000 UTC m=+18.262313450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:02.335549 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:02.335464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:02.335704 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.335591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:02.336019 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:02.336000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:02.336094 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.336075 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:02.336269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:02.336232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:02.336373 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:02.336326 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:04.334247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:04.334391 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:04.334244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:04.334482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:04.334246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:04.334937 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:04.334559 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:06.334713 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:06.334669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:06.335192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:06.334669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:06.335192 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:06.334806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:06.335192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:06.334858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:06.335192 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:06.334955 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:06.335192 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:06.335032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:08.334039 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:08.334001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:08.334451 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:08.334005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:08.334451 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:08.334156 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:08.334451 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:08.334010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:08.334451 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:08.334228 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:08.334451 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:08.334282 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:10.018174 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.018139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:10.018174 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.018180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:10.018676 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.018303 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:10.018676 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.018310 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:10.018676 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.018370 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:26.018353218 +0000 UTC m=+34.223739825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:10.018676 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.018400 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:26.018379467 +0000 UTC m=+34.223766077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:10.119098 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.119068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:10.119289 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.119255 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:10.119289 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.119280 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:10.119407 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.119293 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:10.119407 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.119338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:15:26.119324879 +0000 UTC m=+34.324711499 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:10.333750 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.333724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:10.333879 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.333724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:10.333948 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.333874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:10.334009 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.333981 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:10.334009 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:10.333724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:10.334101 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:10.334081 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:12.334375 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:12.334342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:12.334812 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:12.334454 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:12.334812 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:12.334529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:12.334812 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:12.334629 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:12.334812 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:12.334672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:12.334812 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:12.334730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.334905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:14.335128 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.335412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:14.335469 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.335624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:14.336028 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:14.335722 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:14.447184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.447155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" event={"ID":"8ee91690-c8a4-46e3-acec-d37434cd50c9","Type":"ContainerStarted","Data":"af98acf3bee51eb25c3ec81e676e5a2881aa482f3c495f59345cb4bd63d91df1"} Jul 02 20:15:14.448707 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.448675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tlgf8" event={"ID":"b9c8e8e5-5210-4889-9203-4d0b052a08c4","Type":"ContainerStarted","Data":"cceb0a54531bca854e45b1d296aec76968999bbe6d1aac81067d4ec961b6e80d"} Jul 02 20:15:14.450328 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.450303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hppsm" event={"ID":"29565124-aee4-4ff1-a5a6-f61f8bbf605d","Type":"ContainerStarted","Data":"ab23f97cf676a7e5320760bf6846822bec48382b5c67d332d770802d0e34555e"} Jul 02 20:15:14.452029 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.452004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerStarted","Data":"74bb6d5dcee2b48cf532f54e7172afde5b11c79f3268a1b97b50a054e80bc330"} Jul 02 20:15:14.453563 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.453539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5s5n" event={"ID":"dbcad745-a614-4cca-a2b8-f36c660d9251","Type":"ContainerStarted","Data":"6043a5e614e986e44585bb3fa23d6abe35a483a85497f97df7dece665675eaf3"} Jul 02 20:15:14.454950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.454927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5nfrl" event={"ID":"c555238c-220f-4d0b-ae19-5f4fa5095ba1","Type":"ContainerStarted","Data":"1d37240e1a1e179c1dfefddc24cd319d5ce34398db77f327fdba12a19d21fb69"} Jul 02 20:15:14.456579 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.456558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wvsc6" event={"ID":"60ae01ac-7f49-40e6-a2a7-7faabd755c21","Type":"ContainerStarted","Data":"5edf14064ff12e91f114e8fb4d610f85cc61f8a9773e9c6d8912c3e28cacdf06"} Jul 02 20:15:14.462600 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.462563 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tlgf8" podStartSLOduration=3.670962319 podStartE2EDuration="22.462551856s" totalImagesPullingTime="18.791589537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.462067908 +0000 UTC m=+22.667454537" watchObservedRunningTime="2026-07-02 20:15:14.462551856 +0000 UTC m=+22.667938484" Jul 02 20:15:14.478499 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.478456 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wvsc6" podStartSLOduration=3.456106599 podStartE2EDuration="22.478442776s" totalImagesPullingTime="19.022336177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.478342472 +0000 UTC m=+22.683729101" watchObservedRunningTime="2026-07-02 20:15:14.478442776 +0000 UTC m=+22.683829405" Jul 02 20:15:14.493063 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.493013 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5nfrl" podStartSLOduration=3.529929504 podStartE2EDuration="22.492998661s" totalImagesPullingTime="18.963069157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.49271671 +0000 UTC m=+22.698103339" watchObservedRunningTime="2026-07-02 20:15:14.492998661 +0000 UTC m=+22.698385289" Jul 02 20:15:14.529558 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.529450 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r5s5n" podStartSLOduration=7.536309908 podStartE2EDuration="22.529428855s" totalImagesPullingTime="14.993118947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.528912981 +0000 UTC m=+22.734299625" watchObservedRunningTime="2026-07-02 20:15:14.529428855 +0000 UTC m=+22.734815485" Jul 02 20:15:14.545707 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.545662 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hppsm" podStartSLOduration=3.574518209 podStartE2EDuration="22.545644761s" totalImagesPullingTime="18.971126552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.545094354 +0000 UTC m=+22.750480982" watchObservedRunningTime="2026-07-02 20:15:14.545644761 +0000 UTC m=+22.751031431" Jul 02 20:15:14.597974 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.597942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5s5n_dbcad745-a614-4cca-a2b8-f36c660d9251/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:14.669500 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:14.669468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5s5n_dbcad745-a614-4cca-a2b8-f36c660d9251/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:15.449094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.448947 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 20:15:15.460100 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.460072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" event={"ID":"8ee91690-c8a4-46e3-acec-d37434cd50c9","Type":"ContainerStarted","Data":"ee7e63c6e101566fd1e2984ed097fde4d60aba4ad514e9c96906d9ffd13d54e0"} Jul 02 20:15:15.462855 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.462821 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:15:15.463167 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463138 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2f85b4d-8523-46da-b6db-d0fba83ed62d" containerID="92bb80b5da3f9bbc0acbf62a0efee44d79ae5e006fd39443b6d2c82b3d7790b4" exitCode=1 Jul 02 20:15:15.463236 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"667d9b092f000e37631cd5bd2571b79d3be6bdb2e683e7456d50124c9ddbeadd"} Jul 02 20:15:15.463322 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"3d16207aba0499f88c5fb7f91e0c3fc6169f15b873f643016be01f8ee04524b9"} Jul 02 20:15:15.463416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463352 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"42d042a960e9ea3e271b629cfd27cc251b898b74616c508f77849ec01c379f90"} Jul 02 20:15:15.463416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"31ee17981e75c95944272f5c5f357b765ccaca50200debf32eb950a31089d939"} Jul 02 20:15:15.463416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerDied","Data":"92bb80b5da3f9bbc0acbf62a0efee44d79ae5e006fd39443b6d2c82b3d7790b4"} Jul 02 20:15:15.463514 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.463439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"97767d8eee08fda24514fbf07c96c55f1249fd9c7741f3989e13bd40bfff8217"} Jul 02 20:15:15.464583 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.464565 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="74bb6d5dcee2b48cf532f54e7172afde5b11c79f3268a1b97b50a054e80bc330" exitCode=0 Jul 02 20:15:15.464679 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:15.464660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"74bb6d5dcee2b48cf532f54e7172afde5b11c79f3268a1b97b50a054e80bc330"} Jul 02 20:15:16.240960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.240680 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T20:15:15.449086541Z","UUID":"a6799f3d-9fbb-417c-ba13-c07ffcad69bb","Handler":null,"Name":"","Endpoint":""} Jul 02 20:15:16.242261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.242240 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 20:15:16.242374 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.242284 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 20:15:16.243810 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.243571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:15:16.244167 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.244151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:15:16.334069 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.334038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:16.334249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.334038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:16.334249 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:16.334174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:16.334351 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:16.334275 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:16.334351 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.334044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:16.334418 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:16.334374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:16.468384 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.468346 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jshjl" event={"ID":"dd31db36-e500-4131-8350-76ef7845ded0","Type":"ContainerStarted","Data":"977da9fb5545a882ca8bd50cf1162772459c5f8483ff279818b2d62f4d652b92"} Jul 02 20:15:16.468807 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.468752 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:15:16.469250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.469222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5s5n" Jul 02 20:15:16.484217 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:16.484165 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jshjl" podStartSLOduration=5.525497714 podStartE2EDuration="24.48414959s" totalImagesPullingTime="18.958651876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:16.483596613 +0000 UTC m=+24.688983263" watchObservedRunningTime="2026-07-02 20:15:16.48414959 +0000 UTC m=+24.689536222" Jul 02 20:15:17.472682 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:17.472644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" event={"ID":"8ee91690-c8a4-46e3-acec-d37434cd50c9","Type":"ContainerStarted","Data":"36d3f450e89baf4b379f48823ac21a68380e9bc9cd88a3dd461ca204793ea4da"} Jul 02 20:15:17.476156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:17.476130 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:15:17.476604 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:17.476557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"10d9e74f9c4c9acfdc968bb665c6677391346b63ff4d4c73d786c22aee1c5480"} Jul 02 20:15:17.491656 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:17.491608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsh6q" podStartSLOduration=4.608537039 podStartE2EDuration="25.491587514s" totalImagesPullingTime="20.883050475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:17.491176825 +0000 UTC m=+25.696563476" watchObservedRunningTime="2026-07-02 20:15:17.491587514 +0000 UTC m=+25.696974143" Jul 02 20:15:18.333752 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:18.333720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:18.333955 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:18.333860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:18.333955 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:18.333886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:18.333955 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:18.333927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:18.334122 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:18.334002 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:18.334122 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:18.334065 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:20.334094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:20.334012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:20.334738 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:20.334012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:20.334738 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:20.334154 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:20.334738 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:20.334240 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:20.334738 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:20.334012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:20.334738 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:20.334347 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:21.487799 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.487633 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="7cc916d17b9859b24f12566e28cc4484a1a4cfb225d1ac22ff655e32823a58e7" exitCode=0 Jul 02 20:15:21.488355 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.487711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"7cc916d17b9859b24f12566e28cc4484a1a4cfb225d1ac22ff655e32823a58e7"} Jul 02 20:15:21.494458 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.494407 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:15:21.494811 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.494789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"d4ec7728db059ffb477ef95653dd6f64f0f34ee7cac79ec124b54587010f31fa"} Jul 02 20:15:21.495083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.495063 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:21.495148 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.495136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:21.495303 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.495289 2579 scope.go:117] "RemoveContainer" containerID="92bb80b5da3f9bbc0acbf62a0efee44d79ae5e006fd39443b6d2c82b3d7790b4" Jul 02 20:15:21.511062 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:21.511034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:22.335852 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.335626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:22.336050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.335668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:22.336050 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.335958 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:22.336050 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.336007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:22.336050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.335691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:22.336212 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.336082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:22.500155 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.500129 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:15:22.500713 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.500457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" event={"ID":"e2f85b4d-8523-46da-b6db-d0fba83ed62d","Type":"ContainerStarted","Data":"455fb5a33126eb3f852ee13327240d748d444148ab75f0e4d8921664b6924a54"} Jul 02 20:15:22.500884 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.500864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:22.517709 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.517677 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:22.528616 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.528587 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx5qb"] Jul 02 20:15:22.528784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.528710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:22.528875 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.528802 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:22.533060 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.533011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" podStartSLOduration=11.339864149 podStartE2EDuration="30.532995258s" totalImagesPullingTime="19.193131109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:22.531176016 +0000 UTC m=+30.736562665" watchObservedRunningTime="2026-07-02 20:15:22.532995258 +0000 UTC m=+30.738381887" Jul 02 20:15:22.533889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.533832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twfrq"] Jul 02 20:15:22.533977 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.533967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:22.534077 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.534062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:22.541940 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.541909 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8dm9k"] Jul 02 20:15:22.542061 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:22.542014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:22.542116 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:22.542099 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:23.505235 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:23.505203 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="7229dbc3299309a4df866e2c0d220371cd3a9d8ac402f3254625848b0a03e7ac" exitCode=0 Jul 02 20:15:23.505899 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:23.505300 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"7229dbc3299309a4df866e2c0d220371cd3a9d8ac402f3254625848b0a03e7ac"} Jul 02 20:15:24.336882 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:24.336826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:24.336882 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:24.336880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:24.337092 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:24.336884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:24.337092 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:24.336955 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:24.337092 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:24.337039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:24.337204 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:24.337103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:25.512426 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:25.512392 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="738672c69bcc006c116fcb1c28904c116b12cb8965f2d99e733969d256edbd7c" exitCode=0 Jul 02 20:15:25.512908 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:25.512438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"738672c69bcc006c116fcb1c28904c116b12cb8965f2d99e733969d256edbd7c"} Jul 02 20:15:26.032784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.032691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:26.032784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.032732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:26.032987 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.032867 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:26.032987 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.032870 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:26.032987 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.032920 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret podName:b6a8c413-e177-4d35-95d7-43f15b643628 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:58.032906938 +0000 UTC m=+66.238293545 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret") pod "global-pull-secret-syncer-bx5qb" (UID: "b6a8c413-e177-4d35-95d7-43f15b643628") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:26.032987 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.032934 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:58.032928073 +0000 UTC m=+66.238314680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:26.134071 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.134036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:26.134270 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.134228 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:26.134270 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.134259 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:26.134270 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.134271 2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhj6h for pod openshift-network-diagnostics/network-check-target-twfrq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:26.134422 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.134338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h podName:eaadd0cd-d131-4391-9c8d-948c5f10af2d nodeName:}" failed. No retries permitted until 2026-07-02 20:15:58.134317673 +0000 UTC m=+66.339704284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhj6h" (UniqueName: "kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h") pod "network-check-target-twfrq" (UID: "eaadd0cd-d131-4391-9c8d-948c5f10af2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:26.337169 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.337090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:26.337305 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.337177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:26.337305 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.337181 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfrq" podUID="eaadd0cd-d131-4391-9c8d-948c5f10af2d" Jul 02 20:15:26.337305 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.337275 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:15:26.337305 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:26.337298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:26.337431 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:26.337346 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bx5qb" podUID="b6a8c413-e177-4d35-95d7-43f15b643628" Jul 02 20:15:28.094397 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.094126 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-106.ec2.internal" event="NodeReady" Jul 02 20:15:28.094865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.094489 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 20:15:28.145625 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.145587 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v7fwz"] Jul 02 20:15:28.149004 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.148985 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6jvl"] Jul 02 20:15:28.149129 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.149114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.151613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.151577 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 20:15:28.151752 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.151731 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 20:15:28.151971 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.151956 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7swmr\"" Jul 02 20:15:28.152073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.152044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 20:15:28.152557 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.152540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.155187 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.155158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 20:15:28.155295 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.155279 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzb4r\"" Jul 02 20:15:28.155362 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.155319 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 20:15:28.162611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.162257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v7fwz"] Jul 02 20:15:28.162765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.162618 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6jvl"] Jul 02 20:15:28.252663 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120775c-f933-4bbb-911f-eca81d360290-tmp-dir\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.252892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wgl\" (UniqueName: \"kubernetes.io/projected/2120775c-f933-4bbb-911f-eca81d360290-kube-api-access-c9wgl\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.252892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c287dd62-61a4-4a48-a46a-148d2728a5f3-cert\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.252892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6h2t\" (UniqueName: \"kubernetes.io/projected/c287dd62-61a4-4a48-a46a-148d2728a5f3-kube-api-access-l6h2t\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.252892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.253038 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.252934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2120775c-f933-4bbb-911f-eca81d360290-config-volume\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.333998 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.333967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:28.334190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.333967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:28.334190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.334110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:28.337775 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.337748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:15:28.338007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.337992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 20:15:28.338102 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.338088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2jpl6\"" Jul 02 20:15:28.338219 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.338205 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 20:15:28.338282 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.338268 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:15:28.338386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.338372 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8kznm\"" Jul 02 20:15:28.353656 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6h2t\" (UniqueName: \"kubernetes.io/projected/c287dd62-61a4-4a48-a46a-148d2728a5f3-kube-api-access-l6h2t\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.353786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.353786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2120775c-f933-4bbb-911f-eca81d360290-config-volume\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.353786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120775c-f933-4bbb-911f-eca81d360290-tmp-dir\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.353786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wgl\" (UniqueName: \"kubernetes.io/projected/2120775c-f933-4bbb-911f-eca81d360290-kube-api-access-c9wgl\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.353786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.353771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c287dd62-61a4-4a48-a46a-148d2728a5f3-cert\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.354128 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:28.353803 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:28.354128 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:28.353887 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:28.853863611 +0000 UTC m=+37.059250221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:28.354128 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.354107 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120775c-f933-4bbb-911f-eca81d360290-tmp-dir\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.354360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.354328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2120775c-f933-4bbb-911f-eca81d360290-config-volume\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.358136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.358114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c287dd62-61a4-4a48-a46a-148d2728a5f3-cert\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.364995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.364966 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wgl\" (UniqueName: \"kubernetes.io/projected/2120775c-f933-4bbb-911f-eca81d360290-kube-api-access-c9wgl\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.365205 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.365187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6h2t\" (UniqueName: \"kubernetes.io/projected/c287dd62-61a4-4a48-a46a-148d2728a5f3-kube-api-access-l6h2t\") pod \"ingress-canary-v7fwz\" (UID: \"c287dd62-61a4-4a48-a46a-148d2728a5f3\") " pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.460029 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.459997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v7fwz" Jul 02 20:15:28.648332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.648299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v7fwz"] Jul 02 20:15:28.652137 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:15:28.652109 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc287dd62_61a4_4a48_a46a_148d2728a5f3.slice/crio-7206db1265301ca8255e5bdd0bc384fff5617269dd91ff6fa5054b4d158897f9 WatchSource:0}: Error finding container 7206db1265301ca8255e5bdd0bc384fff5617269dd91ff6fa5054b4d158897f9: Status 404 returned error can't find the container with id 7206db1265301ca8255e5bdd0bc384fff5617269dd91ff6fa5054b4d158897f9 Jul 02 20:15:28.858437 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:28.858339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:28.858609 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:28.858501 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:28.858609 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:28.858568 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:29.858551568 +0000 UTC m=+38.063938179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:29.525072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:29.525032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v7fwz" event={"ID":"c287dd62-61a4-4a48-a46a-148d2728a5f3","Type":"ContainerStarted","Data":"7206db1265301ca8255e5bdd0bc384fff5617269dd91ff6fa5054b4d158897f9"} Jul 02 20:15:29.867620 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:29.867124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:29.867620 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:29.867288 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:29.867620 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:29.867338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:31.867325017 +0000 UTC m=+40.072711624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:31.885344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:31.885121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:31.885677 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:31.885487 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:31.885677 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:31.885546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:35.885531456 +0000 UTC m=+44.090918063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:32.535357 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:32.535322 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="1766649603b41e2c91d0c30b23bc3445c6188aaa24cebc1f7cdcf2b36223f1e6" exitCode=0 Jul 02 20:15:32.535524 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:32.535412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"1766649603b41e2c91d0c30b23bc3445c6188aaa24cebc1f7cdcf2b36223f1e6"} Jul 02 20:15:32.536820 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:32.536796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v7fwz" event={"ID":"c287dd62-61a4-4a48-a46a-148d2728a5f3","Type":"ContainerStarted","Data":"ededc2c24144363792e5730ee379b6de568fca1ded54c19e844ed2c01bcd8e33"} Jul 02 20:15:32.571247 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:32.571085 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v7fwz" podStartSLOduration=1.376379711 podStartE2EDuration="4.571067327s" totalImagesPullingTime="3.194687616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:32.570476043 +0000 UTC m=+40.775862675" watchObservedRunningTime="2026-07-02 20:15:32.571067327 +0000 UTC m=+40.776453956" Jul 02 20:15:33.542074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:33.542038 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c032036-fc2d-4dc2-a98c-6bc734d95c38" containerID="8b859e692e461ef28314fd39731a95f5b3408d157d6ab2b58668bbbdccce5c74" exitCode=0 Jul 02 20:15:33.542448 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:33.542126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerDied","Data":"8b859e692e461ef28314fd39731a95f5b3408d157d6ab2b58668bbbdccce5c74"} Jul 02 20:15:34.547083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:34.547046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" event={"ID":"7c032036-fc2d-4dc2-a98c-6bc734d95c38","Type":"ContainerStarted","Data":"d6ac7fbf72be4ad9b49a55ad32767f9f00a59f3f021844aace1a3679b5a45d1e"} Jul 02 20:15:34.569116 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:34.569063 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s4hd7" podStartSLOduration=9.522303311 podStartE2EDuration="42.56904641s" totalImagesPullingTime="33.046743099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:34.567734505 +0000 UTC m=+42.773121133" watchObservedRunningTime="2026-07-02 20:15:34.56904641 +0000 UTC m=+42.774433039" Jul 02 20:15:35.914570 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:35.914522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:35.914992 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:35.914643 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:35.914992 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:35.914696 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:43.914683172 +0000 UTC m=+52.120069779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:43.971477 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:43.971438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:43.971966 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:43.971613 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:43.971966 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:43.971680 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:59.97166275 +0000 UTC m=+68.177049357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:15:54.518784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:54.518754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mrg6r" Jul 02 20:15:58.066375 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.066338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:15:58.066375 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.066377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:58.069547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.069515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:15:58.069677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.069609 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:15:58.076906 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:58.076886 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:15:58.077000 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:58.076962 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:17:02.076944897 +0000 UTC m=+130.282331517 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : secret "metrics-daemon-secret" not found Jul 02 20:15:58.080020 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.080001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b6a8c413-e177-4d35-95d7-43f15b643628-original-pull-secret\") pod \"global-pull-secret-syncer-bx5qb\" (UID: \"b6a8c413-e177-4d35-95d7-43f15b643628\") " pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:58.167690 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.167649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:58.170863 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.170828 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 20:15:58.180614 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.180589 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 20:15:58.191544 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.191506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhj6h\" (UniqueName: \"kubernetes.io/projected/eaadd0cd-d131-4391-9c8d-948c5f10af2d-kube-api-access-zhj6h\") pod \"network-check-target-twfrq\" (UID: \"eaadd0cd-d131-4391-9c8d-948c5f10af2d\") " pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:58.348478 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.348395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8kznm\"" Jul 02 20:15:58.351475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.351457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bx5qb" Jul 02 20:15:58.356143 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.356123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:15:58.480660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.480629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bx5qb"] Jul 02 20:15:58.484366 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:15:58.484339 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a8c413_e177_4d35_95d7_43f15b643628.slice/crio-b72cf378d17c79608f5253bcb960205b05fa27623454a9c0b7e3cefc2895418f WatchSource:0}: Error finding container b72cf378d17c79608f5253bcb960205b05fa27623454a9c0b7e3cefc2895418f: Status 404 returned error can't find the container with id b72cf378d17c79608f5253bcb960205b05fa27623454a9c0b7e3cefc2895418f Jul 02 20:15:58.497153 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.497124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twfrq"] Jul 02 20:15:58.500539 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:15:58.500515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaadd0cd_d131_4391_9c8d_948c5f10af2d.slice/crio-ac4506e73d2ee97c4d380976d7f8e73ae1077fa2e56b7fcfc80851c8955057c3 WatchSource:0}: Error finding container ac4506e73d2ee97c4d380976d7f8e73ae1077fa2e56b7fcfc80851c8955057c3: Status 404 returned error can't find the container with id ac4506e73d2ee97c4d380976d7f8e73ae1077fa2e56b7fcfc80851c8955057c3 Jul 02 20:15:58.606113 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.606010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twfrq" event={"ID":"eaadd0cd-d131-4391-9c8d-948c5f10af2d","Type":"ContainerStarted","Data":"ac4506e73d2ee97c4d380976d7f8e73ae1077fa2e56b7fcfc80851c8955057c3"} Jul 02 20:15:58.607138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:58.607109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx5qb" event={"ID":"b6a8c413-e177-4d35-95d7-43f15b643628","Type":"ContainerStarted","Data":"b72cf378d17c79608f5253bcb960205b05fa27623454a9c0b7e3cefc2895418f"} Jul 02 20:15:59.979522 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:15:59.979487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:15:59.980118 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:59.979645 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:59.980118 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:15:59.979720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:16:31.979699494 +0000 UTC m=+100.185086102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:16:03.622707 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:03.622665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twfrq" event={"ID":"eaadd0cd-d131-4391-9c8d-948c5f10af2d","Type":"ContainerStarted","Data":"9443a9355752385a604fce49d704896fbe2134d562e5b3fe2e4b300e9c48a728"} Jul 02 20:16:03.623180 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:03.622780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:16:03.624083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:03.624059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bx5qb" event={"ID":"b6a8c413-e177-4d35-95d7-43f15b643628","Type":"ContainerStarted","Data":"08c6d96aeb936f22620a3a6aade4bb5097aab0c1380a6cfa40fdffe0274f34b8"} Jul 02 20:16:03.637739 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:03.637692 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-twfrq" podStartSLOduration=67.18979505 podStartE2EDuration="1m11.637677842s" totalImagesPullingTime="4.447882792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:03.637025811 +0000 UTC m=+71.842412440" watchObservedRunningTime="2026-07-02 20:16:03.637677842 +0000 UTC m=+71.843064471" Jul 02 20:16:03.651028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:03.650980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bx5qb" podStartSLOduration=67.184259522 podStartE2EDuration="1m11.650962986s" totalImagesPullingTime="4.466703464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:03.650220848 +0000 UTC m=+71.855607492" watchObservedRunningTime="2026-07-02 20:16:03.650962986 +0000 UTC m=+71.856349592" Jul 02 20:16:07.267520 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.267487 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5"] Jul 02 20:16:07.276660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.276636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.279693 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.279669 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 02 20:16:07.280760 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 02 20:16:07.280885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280783 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 02 20:16:07.280885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280791 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 20:16:07.280885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 20:16:07.280885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280749 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 02 20:16:07.281107 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.280778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 20:16:07.282984 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.282957 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5"] Jul 02 20:16:07.327797 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.327761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.327797 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.327796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.328082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.327812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.328082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.327898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.328082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.327930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl2rf\" (UniqueName: \"kubernetes.io/projected/78167972-4c57-4c5b-81f4-12bd058431b9-kube-api-access-wl2rf\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.328082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.328013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78167972-4c57-4c5b-81f4-12bd058431b9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428359 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428359 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl2rf\" (UniqueName: \"kubernetes.io/projected/78167972-4c57-4c5b-81f4-12bd058431b9-kube-api-access-wl2rf\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428585 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78167972-4c57-4c5b-81f4-12bd058431b9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428585 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428585 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.428585 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.428501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.429122 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.429095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78167972-4c57-4c5b-81f4-12bd058431b9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.431032 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.431008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.431146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.431048 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.431146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.431086 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-ca\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.431362 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.431341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78167972-4c57-4c5b-81f4-12bd058431b9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.436855 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.436817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl2rf\" (UniqueName: \"kubernetes.io/projected/78167972-4c57-4c5b-81f4-12bd058431b9-kube-api-access-wl2rf\") pod \"cluster-proxy-proxy-agent-58df68f6ff-gq2q5\" (UID: \"78167972-4c57-4c5b-81f4-12bd058431b9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.596230 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.596116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:16:07.717821 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:07.717792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5"] Jul 02 20:16:07.720828 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:16:07.720799 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78167972_4c57_4c5b_81f4_12bd058431b9.slice/crio-e8f995ee964161d657c2511023049771a05c570ca5b5913d7454a87ebb0ada33 WatchSource:0}: Error finding container e8f995ee964161d657c2511023049771a05c570ca5b5913d7454a87ebb0ada33: Status 404 returned error can't find the container with id e8f995ee964161d657c2511023049771a05c570ca5b5913d7454a87ebb0ada33 Jul 02 20:16:08.638799 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:08.638760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerStarted","Data":"e8f995ee964161d657c2511023049771a05c570ca5b5913d7454a87ebb0ada33"} Jul 02 20:16:12.652072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:12.651959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerStarted","Data":"c351efb241a31285b78350a2a18b700538ac413ecaf5f9be793014116c7625b6"} Jul 02 20:16:12.652072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:12.652002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerStarted","Data":"4153d9e974f7b9bee1789e6302eadaced50607d68eaac9a710d3613d635e8436"} Jul 02 20:16:12.652072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:12.652015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerStarted","Data":"cb8b49a9487eb1ca6124c5eaec4b868313f068e5db60711f062be94c97953ca8"} Jul 02 20:16:12.669925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:12.669873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" podStartSLOduration=1.665171413 podStartE2EDuration="5.669858192s" totalImagesPullingTime="4.004686779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:12.668413596 +0000 UTC m=+80.873800229" watchObservedRunningTime="2026-07-02 20:16:12.669858192 +0000 UTC m=+80.875244845" Jul 02 20:16:31.996257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:31.996217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:16:31.996726 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:16:31.996364 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:16:31.996726 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:16:31.996434 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:17:35.99641915 +0000 UTC m=+164.201805757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:16:34.629816 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:16:34.629778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-twfrq" Jul 02 20:17:02.099760 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:17:02.099713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:17:02.100371 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:02.099923 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:17:02.100371 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:02.100025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:04.100001998 +0000 UTC m=+252.305388620 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : secret "metrics-daemon-secret" not found Jul 02 20:17:31.166412 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:31.166356 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x6jvl" podUID="2120775c-f933-4bbb-911f-eca81d360290" Jul 02 20:17:31.356456 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:31.356419 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:17:31.878159 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:17:31.878130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:17:36.051627 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:17:36.051575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:17:36.052173 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:36.051754 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:17:36.052173 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:17:36.051863 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls podName:2120775c-f933-4bbb-911f-eca81d360290 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:38.051817665 +0000 UTC m=+286.257204286 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls") pod "dns-default-x6jvl" (UID: "2120775c-f933-4bbb-911f-eca81d360290") : secret "dns-default-metrics-tls" not found Jul 02 20:17:44.334273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:17:44.334194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:18:53.693438 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.693409 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jllgd"] Jul 02 20:18:53.696201 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.696185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.699249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.699227 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 20:18:53.700327 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.700305 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 20:18:53.700411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.700346 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 20:18:53.700411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.700355 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g7mm9\"" Jul 02 20:18:53.700521 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.700410 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 20:18:53.707635 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.707613 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jllgd"] Jul 02 20:18:53.781377 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.781350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-data-volume\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.781548 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.781386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.781548 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.781404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.781548 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.781486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8wb\" (UniqueName: \"kubernetes.io/projected/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-api-access-vr8wb\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.781672 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.781545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-crio-socket\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-crio-socket\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882224 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-data-volume\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882224 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882224 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882224 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-crio-socket\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882224 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8wb\" (UniqueName: \"kubernetes.io/projected/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-api-access-vr8wb\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.882465 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:53.882250 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:53.882465 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:53.882322 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:53.882465 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:53.882338 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:54.382319706 +0000 UTC m=+242.587706313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : configmap "kube-rbac-proxy" not found Jul 02 20:18:53.882465 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:53.882374 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:54.382359601 +0000 UTC m=+242.587746208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:53.882654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.882499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-data-volume\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:53.894158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:53.894136 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8wb\" (UniqueName: \"kubernetes.io/projected/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-api-access-vr8wb\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:54.386353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:54.386301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:54.386353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:54.386352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:54.386592 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:54.386428 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:54.386592 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:54.386477 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:54.386592 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:54.386502 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:55.386485836 +0000 UTC m=+243.591872442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : configmap "kube-rbac-proxy" not found Jul 02 20:18:54.386592 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:54.386527 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:55.386512419 +0000 UTC m=+243.591899026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:55.394897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:55.394863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:55.395325 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:55.394910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:55.395325 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:55.394920 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:55.395325 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:55.394989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:57.394973292 +0000 UTC m=+245.600359903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : configmap "kube-rbac-proxy" not found Jul 02 20:18:55.395325 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:55.395058 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:55.395325 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:55.395119 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:57.395102758 +0000 UTC m=+245.600489365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:55.559697 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:55.559671 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5nfrl_c555238c-220f-4d0b-ae19-5f4fa5095ba1/dns-node-resolver/0.log" Jul 02 20:18:56.959250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:56.959226 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlgf8_b9c8e8e5-5210-4889-9203-4d0b052a08c4/node-ca/0.log" Jul 02 20:18:57.407725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:57.407701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:57.407964 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:57.407734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:18:57.407964 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:57.407821 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:57.407964 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:57.407906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:01.407891308 +0000 UTC m=+249.613277915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : configmap "kube-rbac-proxy" not found Jul 02 20:18:57.407964 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:57.407824 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:57.408169 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:18:57.407993 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:01.407977122 +0000 UTC m=+249.613363729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:57.759509 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:18:57.759436 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v7fwz_c287dd62-61a4-4a48-a46a-148d2728a5f3/serve-healthcheck-canary/0.log" Jul 02 20:19:01.439578 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:01.439519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:01.440130 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:01.439626 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:19:01.440130 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:01.439687 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:09.439667925 +0000 UTC m=+257.645054546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : configmap "kube-rbac-proxy" not found Jul 02 20:19:01.440130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:01.439680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:01.440130 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:01.439753 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:19:01.440130 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:01.439806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:09.439791245 +0000 UTC m=+257.645177854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:19:04.156899 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:04.156827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:19:04.157316 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:04.156981 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:19:04.157316 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:04.157042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs podName:c1340cfc-2d2c-4048-af47-f70bc4fa1bb2 nodeName:}" failed. No retries permitted until 2026-07-02 20:21:06.157028231 +0000 UTC m=+374.362414838 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs") pod "network-metrics-daemon-8dm9k" (UID: "c1340cfc-2d2c-4048-af47-f70bc4fa1bb2") : secret "metrics-daemon-secret" not found Jul 02 20:19:09.495034 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:09.494991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:09.495034 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:09.495034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:09.495461 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:09.495143 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:19:09.495461 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:09.495202 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls podName:1a68e4a9-eead-4f8a-952d-73a22eaaf7b8 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:25.495186627 +0000 UTC m=+273.700573253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jllgd" (UID: "1a68e4a9-eead-4f8a-952d-73a22eaaf7b8") : secret "insights-runtime-extractor-tls" not found Jul 02 20:19:09.495541 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:09.495476 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:17.072041 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.072005 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6898868755-jm2z9"] Jul 02 20:19:17.077858 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.077822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.080673 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.080650 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 20:19:17.080673 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.080670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7jgs\"" Jul 02 20:19:17.080819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.080650 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 20:19:17.081765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.081737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 20:19:17.085918 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.085902 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 20:19:17.087812 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.087790 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6898868755-jm2z9"] Jul 02 20:19:17.246765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.246733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-certificates\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.246765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.246766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dllb\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-kube-api-access-7dllb\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.246801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-bound-sa-token\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.246883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-image-registry-private-configuration\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.246966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-installation-pull-secrets\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.247038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-tls\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.247065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-trusted-ca\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.247206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.247100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd403086-32d3-42d4-bf9b-e52781fe16c0-ca-trust-extracted\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348168 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-image-registry-private-configuration\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348168 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-installation-pull-secrets\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348336 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-tls\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348336 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-trusted-ca\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd403086-32d3-42d4-bf9b-e52781fe16c0-ca-trust-extracted\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-certificates\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dllb\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-kube-api-access-7dllb\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348568 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-bound-sa-token\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.348806 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.348779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd403086-32d3-42d4-bf9b-e52781fe16c0-ca-trust-extracted\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.349486 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.349464 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-trusted-ca\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.349567 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.349547 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-certificates\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.350627 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.350596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-image-registry-private-configuration\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.350794 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.350777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-registry-tls\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.350872 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.350819 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd403086-32d3-42d4-bf9b-e52781fe16c0-installation-pull-secrets\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.356113 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.356092 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-bound-sa-token\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.356323 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.356303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dllb\" (UniqueName: \"kubernetes.io/projected/cd403086-32d3-42d4-bf9b-e52781fe16c0-kube-api-access-7dllb\") pod \"image-registry-6898868755-jm2z9\" (UID: \"cd403086-32d3-42d4-bf9b-e52781fe16c0\") " pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.386427 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.386406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:17.520551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:17.520521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6898868755-jm2z9"] Jul 02 20:19:17.523738 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:17.523712 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd403086_32d3_42d4_bf9b_e52781fe16c0.slice/crio-ea2cedb66969826a0b022d5cba6f50d5c2c5d4b8410d3d94108bf1c631edf6a3 WatchSource:0}: Error finding container ea2cedb66969826a0b022d5cba6f50d5c2c5d4b8410d3d94108bf1c631edf6a3: Status 404 returned error can't find the container with id ea2cedb66969826a0b022d5cba6f50d5c2c5d4b8410d3d94108bf1c631edf6a3 Jul 02 20:19:18.171920 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:18.171881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6898868755-jm2z9" event={"ID":"cd403086-32d3-42d4-bf9b-e52781fe16c0","Type":"ContainerStarted","Data":"1faab248cb49b21d2384ff19225b1eeb2bf3f2cbe3560fee5a0271a5a6a5f417"} Jul 02 20:19:18.172308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:18.171940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6898868755-jm2z9" event={"ID":"cd403086-32d3-42d4-bf9b-e52781fe16c0","Type":"ContainerStarted","Data":"ea2cedb66969826a0b022d5cba6f50d5c2c5d4b8410d3d94108bf1c631edf6a3"} Jul 02 20:19:18.172308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:18.172017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:18.194199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:18.194156 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6898868755-jm2z9" podStartSLOduration=1.194141428 podStartE2EDuration="1.194141428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:18.193437024 +0000 UTC m=+266.398823664" watchObservedRunningTime="2026-07-02 20:19:18.194141428 +0000 UTC m=+266.399528056" Jul 02 20:19:25.505156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:25.505102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:25.507777 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:25.507743 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1a68e4a9-eead-4f8a-952d-73a22eaaf7b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jllgd\" (UID: \"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8\") " pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:25.804712 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:25.804627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jllgd" Jul 02 20:19:25.928439 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:25.928400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jllgd"] Jul 02 20:19:25.931510 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:25.931481 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a68e4a9_eead_4f8a_952d_73a22eaaf7b8.slice/crio-0535808b46c3a4b5c6c9211d10101c9b77e112ca9aefc9bb85b6217d5e64a33e WatchSource:0}: Error finding container 0535808b46c3a4b5c6c9211d10101c9b77e112ca9aefc9bb85b6217d5e64a33e: Status 404 returned error can't find the container with id 0535808b46c3a4b5c6c9211d10101c9b77e112ca9aefc9bb85b6217d5e64a33e Jul 02 20:19:26.195584 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:26.195549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jllgd" event={"ID":"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8","Type":"ContainerStarted","Data":"0535808b46c3a4b5c6c9211d10101c9b77e112ca9aefc9bb85b6217d5e64a33e"} Jul 02 20:19:27.598234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:27.598181 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" podUID="78167972-4c57-4c5b-81f4-12bd058431b9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 02 20:19:30.211383 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:30.211348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jllgd" event={"ID":"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8","Type":"ContainerStarted","Data":"74257294b1bbae12a943f391f838499b8b2a6c630ecad6d54e6307b1a122f79f"} Jul 02 20:19:30.211756 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:30.211402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jllgd" event={"ID":"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8","Type":"ContainerStarted","Data":"92679f62795728757083ede010a45915a34a29916ad8523787951079536565e1"} Jul 02 20:19:33.181392 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.181365 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g555n"] Jul 02 20:19:33.184464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.184448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.187299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.187282 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 20:19:33.187398 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.187304 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 20:19:33.187766 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.187744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m47xv\"" Jul 02 20:19:33.187902 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.187886 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 20:19:33.188028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.188014 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 20:19:33.189093 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.189077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 20:19:33.189194 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.189085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 20:19:33.222089 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.222063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jllgd" event={"ID":"1a68e4a9-eead-4f8a-952d-73a22eaaf7b8","Type":"ContainerStarted","Data":"81777a2b732d9c7947901e71192e05f2ebad8b3c2ad09a16d1da7b2600822e45"} Jul 02 20:19:33.239426 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.239383 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jllgd" podStartSLOduration=33.798761814 podStartE2EDuration="40.239369334s" totalImagesPullingTime="6.44060752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:18:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:33.238350547 +0000 UTC m=+281.443737178" watchObservedRunningTime="2026-07-02 20:19:33.239369334 +0000 UTC m=+281.444755941" Jul 02 20:19:33.264108 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-wtmp\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264196 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-root\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264196 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-textfile\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264281 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264314 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-sys\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264314 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr92\" (UniqueName: \"kubernetes.io/projected/f46ad247-2f41-40e6-a6c1-40e05ac7323d-kube-api-access-lrr92\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-metrics-client-ca\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.264420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.264379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365237 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-sys\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr92\" (UniqueName: \"kubernetes.io/projected/f46ad247-2f41-40e6-a6c1-40e05ac7323d-kube-api-access-lrr92\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-metrics-client-ca\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-sys\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365388 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:33.365349 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:33.365402 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls podName:f46ad247-2f41-40e6-a6c1-40e05ac7323d nodeName:}" failed. No retries permitted until 2026-07-02 20:19:33.865388308 +0000 UTC m=+282.070774915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls") pod "node-exporter-g555n" (UID: "f46ad247-2f41-40e6-a6c1-40e05ac7323d") : secret "node-exporter-tls" not found Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-wtmp\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-root\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-textfile\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365524 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-root\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-wtmp\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365920 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365790 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-textfile\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.365966 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.366001 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.365947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f46ad247-2f41-40e6-a6c1-40e05ac7323d-metrics-client-ca\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.367926 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.367910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.375014 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.374992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr92\" (UniqueName: \"kubernetes.io/projected/f46ad247-2f41-40e6-a6c1-40e05ac7323d-kube-api-access-lrr92\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.870108 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.870074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:33.872477 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:33.872456 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f46ad247-2f41-40e6-a6c1-40e05ac7323d-node-exporter-tls\") pod \"node-exporter-g555n\" (UID: \"f46ad247-2f41-40e6-a6c1-40e05ac7323d\") " pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:34.094629 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.094563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g555n" Jul 02 20:19:34.106240 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:34.106182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46ad247_2f41_40e6_a6c1_40e05ac7323d.slice/crio-49544912614d0be8e3421b21a35728543d0056564e44286422c3bfd9c5c422c7 WatchSource:0}: Error finding container 49544912614d0be8e3421b21a35728543d0056564e44286422c3bfd9c5c422c7: Status 404 returned error can't find the container with id 49544912614d0be8e3421b21a35728543d0056564e44286422c3bfd9c5c422c7 Jul 02 20:19:34.230095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.230004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g555n" event={"ID":"f46ad247-2f41-40e6-a6c1-40e05ac7323d","Type":"ContainerStarted","Data":"49544912614d0be8e3421b21a35728543d0056564e44286422c3bfd9c5c422c7"} Jul 02 20:19:34.236047 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.236023 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:19:34.240458 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.240443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fpssm\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243146 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243185 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 20:19:34.243268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243220 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 20:19:34.243726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.243639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 20:19:34.249608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.249591 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 20:19:34.254089 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.254067 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:19:34.272047 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272233 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272233 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272293 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272409 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272525 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272560 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncrj\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272624 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.272624 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.272615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373539 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.373711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374314 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.373936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncrj\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374523 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.374504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.374540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.374569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.374599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.374948 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.374923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.376386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.376365 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.376695 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.376674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.377215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.377188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.377369 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.377351 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.377780 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.377754 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.377907 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.377788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.377907 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.377817 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.378158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.378133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.378268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.378246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.378709 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.378694 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.379435 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.379415 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.383068 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.383048 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncrj\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj\") pod \"alertmanager-main-0\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.549512 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.549429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:19:34.681052 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:34.681030 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:19:34.682884 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:34.682859 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354d905c_7401_4f61_8344_2656004b2c6f.slice/crio-2dc697954c47a7d116199fb9adc949aa9ea3fa018238d7ea4fb6b8e50f341c83 WatchSource:0}: Error finding container 2dc697954c47a7d116199fb9adc949aa9ea3fa018238d7ea4fb6b8e50f341c83: Status 404 returned error can't find the container with id 2dc697954c47a7d116199fb9adc949aa9ea3fa018238d7ea4fb6b8e50f341c83 Jul 02 20:19:34.878880 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:34.878801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x6jvl" podUID="2120775c-f933-4bbb-911f-eca81d360290" Jul 02 20:19:35.234172 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:35.234145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"2dc697954c47a7d116199fb9adc949aa9ea3fa018238d7ea4fb6b8e50f341c83"} Jul 02 20:19:35.234456 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:35.234155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:36.238545 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:36.238456 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="6c7cb0364a5413aed99b1992df3e8306a89eccc1ad9c2db7ef14aea2a5d30d1e" exitCode=0 Jul 02 20:19:36.239002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:36.238542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"6c7cb0364a5413aed99b1992df3e8306a89eccc1ad9c2db7ef14aea2a5d30d1e"} Jul 02 20:19:36.240203 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:36.240186 2579 generic.go:356] "Generic (PLEG): container finished" podID="f46ad247-2f41-40e6-a6c1-40e05ac7323d" containerID="de18a6b8a293433f5bd20c6307554e30ff56d0730f3899741adb51226b122fa9" exitCode=0 Jul 02 20:19:36.240268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:36.240251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g555n" event={"ID":"f46ad247-2f41-40e6-a6c1-40e05ac7323d","Type":"ContainerDied","Data":"de18a6b8a293433f5bd20c6307554e30ff56d0730f3899741adb51226b122fa9"} Jul 02 20:19:37.246081 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.246046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g555n" event={"ID":"f46ad247-2f41-40e6-a6c1-40e05ac7323d","Type":"ContainerStarted","Data":"49116bd84fde1b179ded9b2ab197db423a5796277ddaa61849d790963537aa68"} Jul 02 20:19:37.246451 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.246119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g555n" event={"ID":"f46ad247-2f41-40e6-a6c1-40e05ac7323d","Type":"ContainerStarted","Data":"7e0634b13aecad2589a42740103387826d7c6882febfba13711a0d1c4885dcde"} Jul 02 20:19:37.265592 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.265550 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g555n" podStartSLOduration=3.097616421 podStartE2EDuration="4.265537424s" totalImagesPullingTime="1.097736666s" totalInitContainerRuntime="70.184337ms" isStatefulPod=true podCreationTimestamp="2026-07-02 20:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:37.264226835 +0000 UTC m=+285.469613466" watchObservedRunningTime="2026-07-02 20:19:37.265537424 +0000 UTC m=+285.470924052" Jul 02 20:19:37.390575 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.390543 2579 patch_prober.go:28] interesting pod/image-registry-6898868755-jm2z9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 20:19:37.390725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.390594 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6898868755-jm2z9" podUID="cd403086-32d3-42d4-bf9b-e52781fe16c0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 20:19:37.597769 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:37.597678 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" podUID="78167972-4c57-4c5b-81f4-12bd058431b9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 02 20:19:38.103701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:38.103656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:38.106370 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:38.106334 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2120775c-f933-4bbb-911f-eca81d360290-metrics-tls\") pod \"dns-default-x6jvl\" (UID: \"2120775c-f933-4bbb-911f-eca81d360290\") " pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:38.237539 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:38.237506 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzb4r\"" Jul 02 20:19:38.245650 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:38.245621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:38.416291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:38.416269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6jvl"] Jul 02 20:19:38.418164 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:38.418140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2120775c_f933_4bbb_911f_eca81d360290.slice/crio-99fcb59fa22e76bc553166ddab50fda2b3af71f7f49d0d5c576a66ce53817bc9 WatchSource:0}: Error finding container 99fcb59fa22e76bc553166ddab50fda2b3af71f7f49d0d5c576a66ce53817bc9: Status 404 returned error can't find the container with id 99fcb59fa22e76bc553166ddab50fda2b3af71f7f49d0d5c576a66ce53817bc9 Jul 02 20:19:39.180368 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.180332 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6898868755-jm2z9" Jul 02 20:19:39.255731 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.255697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"31b5d7e85837cf95319173b04bb0f8101464af98d6086eccd211a11cd4d1cc3d"} Jul 02 20:19:39.255934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.255771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"fabcfaa44ba33e13ac4dd09f01322abc4a0b562ac01d2c304bcd263aa1083d15"} Jul 02 20:19:39.255934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.255806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"fe0251c67bb406944e78c0db595387b23fbe0baf6cf8a248b59701d9989749b5"} Jul 02 20:19:39.255934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.255852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"69ca04186bbc2cb90404df287e2dd9164a85035459ee2d53ad2b048f9965a602"} Jul 02 20:19:39.255934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.255882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"6f75fdd43567be9da2c25936591fa036a152f1a7f28c5e4ddabd5490f21ec137"} Jul 02 20:19:39.256697 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.256678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6jvl" event={"ID":"2120775c-f933-4bbb-911f-eca81d360290","Type":"ContainerStarted","Data":"99fcb59fa22e76bc553166ddab50fda2b3af71f7f49d0d5c576a66ce53817bc9"} Jul 02 20:19:39.354217 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.354183 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:19:39.357882 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.357859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.360596 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 20:19:39.360596 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360520 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 20:19:39.360759 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 20:19:39.360759 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k9bwf\"" Jul 02 20:19:39.360988 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360971 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 20:19:39.360988 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.360984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 20:19:39.361360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f271ek447phh9\"" Jul 02 20:19:39.361431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 20:19:39.361960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361767 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 20:19:39.361960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361896 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 20:19:39.361960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361926 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 20:19:39.361960 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.361925 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 20:19:39.362181 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.362002 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 20:19:39.362228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.362202 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 20:19:39.362667 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.362649 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 20:19:39.365086 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.365034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 20:19:39.368075 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.368050 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 20:19:39.378082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.376955 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:19:39.413601 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.413726 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.413956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.414363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.414218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515246 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515246 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515246 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.515786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.515772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.516628 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.516327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.516628 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.516469 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.516922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.516779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.517031 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.516958 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.517213 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.517140 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.518454 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.518427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.518692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.518666 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.519146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.519124 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.519548 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.519523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.520073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.520055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.520770 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.520741 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.521023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.520998 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.521277 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.521246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.521686 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.521658 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.521784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.521687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.522112 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.522078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.522448 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.522404 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.522976 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.522954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.523736 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.523703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.524117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.524096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77\") pod \"prometheus-k8s-0\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.669944 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.669917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:39.816094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:39.816058 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:19:40.178496 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:19:40.178462 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a27f2c_c278_4add_9ee5_9d9e1dbe9d5b.slice/crio-902cbbe97a8659d14128f6155d8273771778c77d06aa2e6392731f6a5cc4089e WatchSource:0}: Error finding container 902cbbe97a8659d14128f6155d8273771778c77d06aa2e6392731f6a5cc4089e: Status 404 returned error can't find the container with id 902cbbe97a8659d14128f6155d8273771778c77d06aa2e6392731f6a5cc4089e Jul 02 20:19:40.262175 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:40.262132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"902cbbe97a8659d14128f6155d8273771778c77d06aa2e6392731f6a5cc4089e"} Jul 02 20:19:41.267157 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.267120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6jvl" event={"ID":"2120775c-f933-4bbb-911f-eca81d360290","Type":"ContainerStarted","Data":"ccd0fbb5066851a38611d3812db61950b3cd52b376dfa6a4258674735fda075d"} Jul 02 20:19:41.267566 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.267181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6jvl" event={"ID":"2120775c-f933-4bbb-911f-eca81d360290","Type":"ContainerStarted","Data":"c8f2ef30b6ef4601064b026a84ef87f1080b47ad5753070beb9d8f6b4e1118da"} Jul 02 20:19:41.267566 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.267214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:41.270083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.270062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerStarted","Data":"9b2a105c90d2b0b4ca942f28b9dc44504b6e9ad5e58be8e0bc83711a6df0a6f8"} Jul 02 20:19:41.271459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.271440 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" exitCode=0 Jul 02 20:19:41.271537 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.271468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} Jul 02 20:19:41.286827 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.286667 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x6jvl" podStartSLOduration=251.352118348 podStartE2EDuration="4m13.286653003s" totalImagesPullingTime="1.934534655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:41.286327553 +0000 UTC m=+289.491714204" watchObservedRunningTime="2026-07-02 20:19:41.286653003 +0000 UTC m=+289.492039631" Jul 02 20:19:41.337328 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:41.337279 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.245373249 podStartE2EDuration="7.337265943s" totalImagesPullingTime="5.073548225s" totalInitContainerRuntime="18.344469ms" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:41.336165111 +0000 UTC m=+289.541551774" watchObservedRunningTime="2026-07-02 20:19:41.337265943 +0000 UTC m=+289.542652572" Jul 02 20:19:45.291341 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:45.291308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} Jul 02 20:19:45.291682 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:45.291399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} Jul 02 20:19:47.301283 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.301249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} Jul 02 20:19:47.301659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.301315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} Jul 02 20:19:47.301659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.301341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} Jul 02 20:19:47.301659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.301373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerStarted","Data":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} Jul 02 20:19:47.331005 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.330938 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9314462580000002 podStartE2EDuration="8.330916417s" totalImagesPullingTime="5.377581861s" totalInitContainerRuntime="21.888298ms" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:47.329099211 +0000 UTC m=+295.534485839" watchObservedRunningTime="2026-07-02 20:19:47.330916417 +0000 UTC m=+295.536303046" Jul 02 20:19:47.335347 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:19:47.335319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8dm9k" podUID="c1340cfc-2d2c-4048-af47-f70bc4fa1bb2" Jul 02 20:19:47.597589 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.597496 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" podUID="78167972-4c57-4c5b-81f4-12bd058431b9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 02 20:19:47.597589 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.597586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" Jul 02 20:19:47.598136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.598100 2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c351efb241a31285b78350a2a18b700538ac413ecaf5f9be793014116c7625b6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 02 20:19:47.598252 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:47.598157 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" podUID="78167972-4c57-4c5b-81f4-12bd058431b9" containerName="service-proxy" containerID="cri-o://c351efb241a31285b78350a2a18b700538ac413ecaf5f9be793014116c7625b6" gracePeriod=30 Jul 02 20:19:48.307268 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:48.307235 2579 generic.go:356] "Generic (PLEG): container finished" podID="78167972-4c57-4c5b-81f4-12bd058431b9" containerID="c351efb241a31285b78350a2a18b700538ac413ecaf5f9be793014116c7625b6" exitCode=2 Jul 02 20:19:48.307628 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:48.307309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerDied","Data":"c351efb241a31285b78350a2a18b700538ac413ecaf5f9be793014116c7625b6"} Jul 02 20:19:48.307628 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:48.307351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58df68f6ff-gq2q5" event={"ID":"78167972-4c57-4c5b-81f4-12bd058431b9","Type":"ContainerStarted","Data":"12d5ef26a2fe926281b55c5335a9adc39735a3c4e5d57996bc1e503b3eb217d5"} Jul 02 20:19:49.670371 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:49.670339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:19:51.277302 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:51.277270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6jvl" Jul 02 20:19:52.200587 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:52.200548 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:19:52.201223 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:52.201206 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:19:52.205791 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:52.205776 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 20:19:59.334470 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:19:59.334436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:20:39.670350 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:39.670312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:39.690405 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:39.690375 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:40.505005 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:40.504976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:53.371725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.371644 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:53.372145 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372015 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="alertmanager" containerID="cri-o://6f75fdd43567be9da2c25936591fa036a152f1a7f28c5e4ddabd5490f21ec137" gracePeriod=120 Jul 02 20:20:53.372218 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372123 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-web" containerID="cri-o://fe0251c67bb406944e78c0db595387b23fbe0baf6cf8a248b59701d9989749b5" gracePeriod=120 Jul 02 20:20:53.372218 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372139 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="prom-label-proxy" containerID="cri-o://9b2a105c90d2b0b4ca942f28b9dc44504b6e9ad5e58be8e0bc83711a6df0a6f8" gracePeriod=120 Jul 02 20:20:53.372218 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372152 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy" containerID="cri-o://fabcfaa44ba33e13ac4dd09f01322abc4a0b562ac01d2c304bcd263aa1083d15" gracePeriod=120 Jul 02 20:20:53.372363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372149 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="config-reloader" containerID="cri-o://69ca04186bbc2cb90404df287e2dd9164a85035459ee2d53ad2b048f9965a602" gracePeriod=120 Jul 02 20:20:53.372363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.372087 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-metric" containerID="cri-o://31b5d7e85837cf95319173b04bb0f8101464af98d6086eccd211a11cd4d1cc3d" gracePeriod=120 Jul 02 20:20:53.538529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538490 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="9b2a105c90d2b0b4ca942f28b9dc44504b6e9ad5e58be8e0bc83711a6df0a6f8" exitCode=0 Jul 02 20:20:53.538529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538517 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="fabcfaa44ba33e13ac4dd09f01322abc4a0b562ac01d2c304bcd263aa1083d15" exitCode=0 Jul 02 20:20:53.538529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538523 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="69ca04186bbc2cb90404df287e2dd9164a85035459ee2d53ad2b048f9965a602" exitCode=0 Jul 02 20:20:53.538529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538529 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="6f75fdd43567be9da2c25936591fa036a152f1a7f28c5e4ddabd5490f21ec137" exitCode=0 Jul 02 20:20:53.538792 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"9b2a105c90d2b0b4ca942f28b9dc44504b6e9ad5e58be8e0bc83711a6df0a6f8"} Jul 02 20:20:53.538792 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"fabcfaa44ba33e13ac4dd09f01322abc4a0b562ac01d2c304bcd263aa1083d15"} Jul 02 20:20:53.538792 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"69ca04186bbc2cb90404df287e2dd9164a85035459ee2d53ad2b048f9965a602"} Jul 02 20:20:53.538792 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:53.538705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"6f75fdd43567be9da2c25936591fa036a152f1a7f28c5e4ddabd5490f21ec137"} Jul 02 20:20:54.546617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.546587 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="31b5d7e85837cf95319173b04bb0f8101464af98d6086eccd211a11cd4d1cc3d" exitCode=0 Jul 02 20:20:54.546617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.546613 2579 generic.go:356] "Generic (PLEG): container finished" podID="354d905c-7401-4f61-8344-2656004b2c6f" containerID="fe0251c67bb406944e78c0db595387b23fbe0baf6cf8a248b59701d9989749b5" exitCode=0 Jul 02 20:20:54.547026 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.546623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"31b5d7e85837cf95319173b04bb0f8101464af98d6086eccd211a11cd4d1cc3d"} Jul 02 20:20:54.547026 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.546711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"fe0251c67bb406944e78c0db595387b23fbe0baf6cf8a248b59701d9989749b5"} Jul 02 20:20:54.608239 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.608215 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:54.669221 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669184 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669241 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669302 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669341 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669456 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669487 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669553 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ncrj\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669593 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:54.669655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca\") pod \"354d905c-7401-4f61-8344-2656004b2c6f\" (UID: \"354d905c-7401-4f61-8344-2656004b2c6f\") " Jul 02 20:20:54.670084 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.669994 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.670313 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.670288 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:20:54.670389 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.670319 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:54.673886 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.673544 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume" (OuterVolumeSpecName: "config-volume") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.674984 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.674945 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.675121 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.675102 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj" (OuterVolumeSpecName: "kube-api-access-8ncrj") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "kube-api-access-8ncrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:20:54.678731 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.678697 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out" (OuterVolumeSpecName: "config-out") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:20:54.683099 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.682614 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.684083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.684045 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.692805 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.692765 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.692958 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.692803 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.693293 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.693264 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:20:54.700654 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.700593 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config" (OuterVolumeSpecName: "web-config") pod "354d905c-7401-4f61-8344-2656004b2c6f" (UID: "354d905c-7401-4f61-8344-2656004b2c6f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:54.770954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.770915 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-cluster-tls-config\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.770954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.770946 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncrj\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-kube-api-access-8ncrj\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.770954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.770961 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.770975 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/354d905c-7401-4f61-8344-2656004b2c6f-metrics-client-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.770988 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771002 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-web-config\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771013 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-main-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771025 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-config-out\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771037 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/354d905c-7401-4f61-8344-2656004b2c6f-alertmanager-main-db\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771048 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/354d905c-7401-4f61-8344-2656004b2c6f-tls-assets\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771059 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-config-volume\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:54.771211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:54.771070 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/354d905c-7401-4f61-8344-2656004b2c6f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:55.553742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.553709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"354d905c-7401-4f61-8344-2656004b2c6f","Type":"ContainerDied","Data":"2dc697954c47a7d116199fb9adc949aa9ea3fa018238d7ea4fb6b8e50f341c83"} Jul 02 20:20:55.554140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.553746 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.554140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.553795 2579 scope.go:117] "RemoveContainer" containerID="9b2a105c90d2b0b4ca942f28b9dc44504b6e9ad5e58be8e0bc83711a6df0a6f8" Jul 02 20:20:55.561550 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.561437 2579 scope.go:117] "RemoveContainer" containerID="31b5d7e85837cf95319173b04bb0f8101464af98d6086eccd211a11cd4d1cc3d" Jul 02 20:20:55.568471 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.568454 2579 scope.go:117] "RemoveContainer" containerID="fabcfaa44ba33e13ac4dd09f01322abc4a0b562ac01d2c304bcd263aa1083d15" Jul 02 20:20:55.575169 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.575151 2579 scope.go:117] "RemoveContainer" containerID="fe0251c67bb406944e78c0db595387b23fbe0baf6cf8a248b59701d9989749b5" Jul 02 20:20:55.577070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.577051 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:55.580442 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.580414 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:55.583035 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.583021 2579 scope.go:117] "RemoveContainer" containerID="69ca04186bbc2cb90404df287e2dd9164a85035459ee2d53ad2b048f9965a602" Jul 02 20:20:55.589967 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.589949 2579 scope.go:117] "RemoveContainer" containerID="6f75fdd43567be9da2c25936591fa036a152f1a7f28c5e4ddabd5490f21ec137" Jul 02 20:20:55.597076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.597059 2579 scope.go:117] "RemoveContainer" containerID="6c7cb0364a5413aed99b1992df3e8306a89eccc1ad9c2db7ef14aea2a5d30d1e" Jul 02 20:20:55.607887 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.607866 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:55.608101 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608090 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="config-reloader" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608104 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="config-reloader" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608113 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="prom-label-proxy" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608118 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="prom-label-proxy" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608126 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608131 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608138 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-metric" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608144 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-metric" Jul 02 20:20:55.608151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608152 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-web" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608157 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-web" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608166 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="alertmanager" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608171 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="alertmanager" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608181 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="init-config-reloader" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608185 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="init-config-reloader" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608221 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608230 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="alertmanager" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608236 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="config-reloader" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608241 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-web" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608247 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="kube-rbac-proxy-metric" Jul 02 20:20:55.608411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.608252 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d905c-7401-4f61-8344-2656004b2c6f" containerName="prom-label-proxy" Jul 02 20:20:55.611507 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.611491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.614218 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614200 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614575 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fpssm\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614576 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 20:20:55.614772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.614460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 20:20:55.620586 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.620568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 20:20:55.626912 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.626892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:55.677641 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677641 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-config-out\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-web-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.677868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.677983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96wm\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-kube-api-access-m96wm\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.678001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.678156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.678028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779018 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.778930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779018 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.778980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96wm\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-kube-api-access-m96wm\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779018 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-config-out\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-web-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.779811 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779782 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.780115 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.779955 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.780658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.780630 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2722554-4573-42b4-8a1d-1b75870e7196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.782744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.782603 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.782744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.782621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.782744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.782697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.782744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.782705 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2722554-4573-42b4-8a1d-1b75870e7196-config-out\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.783019 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.782862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-web-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.783070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.783050 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.783434 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.783410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.784493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.784474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.784553 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.784482 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2722554-4573-42b4-8a1d-1b75870e7196-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.787398 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.787379 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96wm\" (UniqueName: \"kubernetes.io/projected/f2722554-4573-42b4-8a1d-1b75870e7196-kube-api-access-m96wm\") pod \"alertmanager-main-0\" (UID: \"f2722554-4573-42b4-8a1d-1b75870e7196\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:55.921723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:55.921689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 20:20:56.054892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:56.054783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 20:20:56.058567 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:20:56.058544 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2722554_4573_42b4_8a1d_1b75870e7196.slice/crio-f4a22d6e239d75a19bfa4c735223437178622366f16b67da73b099836571ddf7 WatchSource:0}: Error finding container f4a22d6e239d75a19bfa4c735223437178622366f16b67da73b099836571ddf7: Status 404 returned error can't find the container with id f4a22d6e239d75a19bfa4c735223437178622366f16b67da73b099836571ddf7 Jul 02 20:20:56.339608 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:56.339532 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354d905c-7401-4f61-8344-2656004b2c6f" path="/var/lib/kubelet/pods/354d905c-7401-4f61-8344-2656004b2c6f/volumes" Jul 02 20:20:56.558876 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:56.558825 2579 generic.go:356] "Generic (PLEG): container finished" podID="f2722554-4573-42b4-8a1d-1b75870e7196" containerID="18e6bf6d36c70616cbcce875bcf096baa2443e668fde6eb711af0013b6ddb273" exitCode=0 Jul 02 20:20:56.559315 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:56.558876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerDied","Data":"18e6bf6d36c70616cbcce875bcf096baa2443e668fde6eb711af0013b6ddb273"} Jul 02 20:20:56.559315 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:56.558951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"f4a22d6e239d75a19bfa4c735223437178622366f16b67da73b099836571ddf7"} Jul 02 20:20:57.572176 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"9d23d625dee4898a028f622b2851d945a7b70e98dff3e875e44de26994925746"} Jul 02 20:20:57.572540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"ffe48ec067416029e73b130c67e5afc5e2ebc5d9f94fd67786236550b91ed674"} Jul 02 20:20:57.572540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"4e6ed08bce71ee184f38ae4c003c176eac0381b62fafe7f64d5f0191da3f2f09"} Jul 02 20:20:57.572540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"08674d4e55df435c4c8ba26a30641b1487c59ab57d01070da9d2863d4f814897"} Jul 02 20:20:57.572540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"7bce2c57356e9d12795869797eeea14c030c63e30423e47aaf9f45b1c94e7d73"} Jul 02 20:20:57.572540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.572416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2722554-4573-42b4-8a1d-1b75870e7196","Type":"ContainerStarted","Data":"743775b8e4316eb3e9bd2120b1b164e0c791384d152057a506a9fbb3076c5c42"} Jul 02 20:20:57.601078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.601030 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.583757631 podStartE2EDuration="2.601013818s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.256187ms" isStatefulPod=false podCreationTimestamp="2026-07-02 20:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:20:57.599271508 +0000 UTC m=+365.804658167" watchObservedRunningTime="2026-07-02 20:20:57.601013818 +0000 UTC m=+365.806400447" Jul 02 20:20:57.703438 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703408 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:57.703725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703695 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="prometheus" containerID="cri-o://a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" gracePeriod=600 Jul 02 20:20:57.703820 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703726 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy" containerID="cri-o://08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" gracePeriod=600 Jul 02 20:20:57.703820 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703700 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" gracePeriod=600 Jul 02 20:20:57.703944 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703731 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="config-reloader" containerID="cri-o://d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" gracePeriod=600 Jul 02 20:20:57.703944 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703715 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="thanos-sidecar" containerID="cri-o://7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" gracePeriod=600 Jul 02 20:20:57.704037 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.703700 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-web" containerID="cri-o://977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" gracePeriod=600 Jul 02 20:20:57.940742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:57.940718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.100136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100088 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100177 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100407 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100500 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100540 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100598 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100630 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100682 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.100704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100738 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100764 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77\") pod \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\" (UID: \"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b\") " Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.100877 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101115 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101294 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.101431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101412 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.101717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101432 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.101717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101532 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.101717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101548 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.101717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101565 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.101938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.101888 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.103248 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.103213 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.103380 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.103245 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:20:58.103896 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.103764 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:58.104190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.104162 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.104390 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.104363 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.104679 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.104654 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77" (OuterVolumeSpecName: "kube-api-access-84j77") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "kube-api-access-84j77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:20:58.104964 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.104939 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.105235 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.105195 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:20:58.105501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.105477 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.105501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.105482 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out" (OuterVolumeSpecName: "config-out") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:20:58.105605 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.105583 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.106054 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.106031 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.106204 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.106178 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config" (OuterVolumeSpecName: "config") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.113950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.113924 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config" (OuterVolumeSpecName: "web-config") pod "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" (UID: "e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202237 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-configmap-metrics-client-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202273 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-tls-assets\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202285 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202298 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202310 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-kube-rbac-proxy\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202321 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-prometheus-k8s-db\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202333 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202346 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202361 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-grpc-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202372 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202382 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-kube-api-access-84j77\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202393 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202406 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-metrics-client-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202417 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202429 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-config-out\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.202658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.202440 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b-web-config\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580223 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580249 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580256 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580264 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580269 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580274 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" exitCode=0 Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} Jul 02 20:20:58.580317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580313 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580456 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} Jul 02 20:20:58.581095 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.580663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b","Type":"ContainerDied","Data":"902cbbe97a8659d14128f6155d8273771778c77d06aa2e6392731f6a5cc4089e"} Jul 02 20:20:58.588421 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.588401 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.595464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.595444 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.602724 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.602701 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.603115 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.603090 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:58.608173 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.608151 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:58.610243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.610227 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.617029 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.617011 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.624399 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.624383 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.631184 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.631169 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.631576 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.631551 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.631666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.631587 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.631666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.631615 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.632016 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.631892 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.632016 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.631923 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.632016 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.631948 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.632294 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.632266 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.632399 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632302 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.632399 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632321 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.632643 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.632626 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.632714 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632651 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.632714 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632673 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.632949 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.632926 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.632994 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632954 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.632994 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632970 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.632994 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.632981 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:58.633226 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.633204 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.633266 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633235 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.633266 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633252 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-thanos" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-thanos" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633277 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="config-reloader" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633284 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="config-reloader" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633297 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="prometheus" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633302 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="prometheus" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633308 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-web" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633314 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-web" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633325 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="init-config-reloader" Jul 02 20:20:58.633332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633334 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="init-config-reloader" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633342 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633347 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633353 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="thanos-sidecar" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633361 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="thanos-sidecar" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633255 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633404 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-thanos" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633413 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="thanos-sidecar" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633418 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy-web" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633426 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="kube-rbac-proxy" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633434 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="prometheus" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633439 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" containerName="config-reloader" Jul 02 20:20:58.633603 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:20:58.633600 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.634059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633618 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.634059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633630 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.634059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633831 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.634059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.633886 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.634190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634116 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.634190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634132 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.634387 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634369 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.634452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634388 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.634644 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634628 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.634689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634645 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.634874 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634854 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.634923 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.634875 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.635084 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635067 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.635123 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635087 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.635369 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635350 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.635415 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635370 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.635588 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635572 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.635635 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635589 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.635817 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635801 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.635878 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.635818 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.636072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636056 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.636106 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636074 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.636309 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636292 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.636360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636310 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.636497 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636481 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.636533 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636498 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.636723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636703 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.636723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636721 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.636947 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636931 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.636987 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.636947 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.637155 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637138 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.637217 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637156 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.637316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.637384 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637345 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.637384 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637366 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.637602 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637577 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.637650 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637605 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.637849 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637804 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.637849 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.637826 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.638132 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638113 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.638132 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638132 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.638316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638299 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.638316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638316 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.638538 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638522 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.638538 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638536 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.638729 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638709 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.638729 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638727 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.639020 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.638994 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.639020 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639014 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.639220 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639202 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.639220 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639220 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.639423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639407 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.639423 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639423 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.639666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639646 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.639666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639664 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.639938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639913 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.640023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.639938 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.640098 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640084 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 20:20:58.640147 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640095 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 20:20:58.640147 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 20:20:58.640343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640314 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.640343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640343 2579 scope.go:117] "RemoveContainer" containerID="6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224" Jul 02 20:20:58.640551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640419 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 20:20:58.640551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f271ek447phh9\"" Jul 02 20:20:58.640551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 20:20:58.640716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 20:20:58.640716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640647 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 20:20:58.640716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 20:20:58.640933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640711 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224"} err="failed to get container status \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": rpc error: code = NotFound desc = could not find container \"6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224\": container with ID starting with 6ceeaee7c76a1e5a7d7b894c5ca5e16cf104b4c9613154a8a274218ba9afd224 not found: ID does not exist" Jul 02 20:20:58.640933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640732 2579 scope.go:117] "RemoveContainer" containerID="08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37" Jul 02 20:20:58.640933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640859 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 20:20:58.640933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640867 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 20:20:58.641081 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.640996 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37"} err="failed to get container status \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": rpc error: code = NotFound desc = could not find container \"08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37\": container with ID starting with 08ea51449e6513beb8856c36703bdf2f158268e112cd697c3a15d6827a821a37 not found: ID does not exist" Jul 02 20:20:58.641081 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641018 2579 scope.go:117] "RemoveContainer" containerID="977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a" Jul 02 20:20:58.641260 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641241 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a"} err="failed to get container status \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": rpc error: code = NotFound desc = could not find container \"977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a\": container with ID starting with 977d7945bac63b6a069c539c9ae4139f3c937b600a69019efa887ce1288b1d9a not found: ID does not exist" Jul 02 20:20:58.641260 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641259 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 20:20:58.641407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641262 2579 scope.go:117] "RemoveContainer" containerID="7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f" Jul 02 20:20:58.641407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 20:20:58.641407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641366 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k9bwf\"" Jul 02 20:20:58.641552 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641505 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 20:20:58.641552 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641504 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f"} err="failed to get container status \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": rpc error: code = NotFound desc = could not find container \"7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f\": container with ID starting with 7f5dfb5b0a947e87d02c9c579d641c1c6cec40b3774e9b2ec85278360c4a941f not found: ID does not exist" Jul 02 20:20:58.641552 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641525 2579 scope.go:117] "RemoveContainer" containerID="d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b" Jul 02 20:20:58.641781 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641758 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b"} err="failed to get container status \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": rpc error: code = NotFound desc = could not find container \"d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b\": container with ID starting with d5c173a47b3f0dbcb881f0c4e1e15aceb341d648c548b422e492cddacdf7cb1b not found: ID does not exist" Jul 02 20:20:58.641781 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.641780 2579 scope.go:117] "RemoveContainer" containerID="a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e" Jul 02 20:20:58.642082 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.642058 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e"} err="failed to get container status \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": rpc error: code = NotFound desc = could not find container \"a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e\": container with ID starting with a20561961e36e7fc37f5bc72b53b9f36d0600b20e00d140d041d465f3f22690e not found: ID does not exist" Jul 02 20:20:58.642173 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.642084 2579 scope.go:117] "RemoveContainer" containerID="c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65" Jul 02 20:20:58.642308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.642286 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65"} err="failed to get container status \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": rpc error: code = NotFound desc = could not find container \"c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65\": container with ID starting with c44f4fd2cf2071221d31cc091c55aa1808afe1747b9b14b30bbf8d4f1b955d65 not found: ID does not exist" Jul 02 20:20:58.643510 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.643489 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 20:20:58.646530 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.646512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 20:20:58.653659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.653638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:58.808001 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.807956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808001 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.807998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl85j\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-kube-api-access-pl85j\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.808617 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.808508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909621 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909775 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909775 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl85j\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-kube-api-access-pl85j\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.909954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.909921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910333 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910429 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910429 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910429 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.910970 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.910796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913102 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913448 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913419 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913505 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913541 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.913972 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.913948 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914043 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914043 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914640 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.914865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.914806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.916019 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.916001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.916322 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.916303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.916658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.916639 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-config\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.917438 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.917420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.918192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.918176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl85j\" (UniqueName: \"kubernetes.io/projected/0a10bc5d-3c3a-4532-b4d8-d93ff115748f-kube-api-access-pl85j\") pod \"prometheus-k8s-0\" (UID: \"0a10bc5d-3c3a-4532-b4d8-d93ff115748f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:58.948318 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:58.948282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:20:59.084929 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:59.084898 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 20:20:59.087783 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:20:59.087759 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a10bc5d_3c3a_4532_b4d8_d93ff115748f.slice/crio-f836a39ac1b49ddb3c0bda11cac53ed8f64991fd95420924ee34c199beb2f726 WatchSource:0}: Error finding container f836a39ac1b49ddb3c0bda11cac53ed8f64991fd95420924ee34c199beb2f726: Status 404 returned error can't find the container with id f836a39ac1b49ddb3c0bda11cac53ed8f64991fd95420924ee34c199beb2f726 Jul 02 20:20:59.585798 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:59.585716 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a10bc5d-3c3a-4532-b4d8-d93ff115748f" containerID="713d998b33424d12c08ecb8a7a61a0a1f4a276dc5a92f746a0b3a4b42715d7c6" exitCode=0 Jul 02 20:20:59.586210 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:59.585808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerDied","Data":"713d998b33424d12c08ecb8a7a61a0a1f4a276dc5a92f746a0b3a4b42715d7c6"} Jul 02 20:20:59.586210 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:20:59.585895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"f836a39ac1b49ddb3c0bda11cac53ed8f64991fd95420924ee34c199beb2f726"} Jul 02 20:21:00.339655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.339624 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b" path="/var/lib/kubelet/pods/e4a27f2c-c278-4add-9ee5-9d9e1dbe9d5b/volumes" Jul 02 20:21:00.594023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.593930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"bfefa8d4fef95641ca3fb5a1937356979d2f0ae019693c82a520fa363a2c74dd"} Jul 02 20:21:00.594023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.593994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"0b6a6b3f50e73022e1c89b57fc3c0390cfe369508990c0bf584fd675789bf442"} Jul 02 20:21:00.594439 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.594036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"4e1ddf613a98a7234b7f05773762706eb7ea5f81cef935a5c396f342ffc361e2"} Jul 02 20:21:00.594439 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.594068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"07f9131f099a8eac17a72026611269c5a61e13ea8f78a2c2c26f2da181c01fbe"} Jul 02 20:21:00.594439 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.594095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"3b7338073a662c33c213c698c7d795782f98b93fa7afacc5b9b58cb4ecf21ea2"} Jul 02 20:21:00.594439 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.594135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a10bc5d-3c3a-4532-b4d8-d93ff115748f","Type":"ContainerStarted","Data":"79782b895c24fe5e28ff00719cc793100d6e9bebe6c039fcee2d06d6fca82cf6"} Jul 02 20:21:00.623889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:00.623823 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6029872210000002 podStartE2EDuration="2.62380675s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.819529ms" isStatefulPod=false podCreationTimestamp="2026-07-02 20:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:21:00.621463935 +0000 UTC m=+368.826850579" watchObservedRunningTime="2026-07-02 20:21:00.62380675 +0000 UTC m=+368.829193378" Jul 02 20:21:03.948486 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:03.948434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:21:06.174936 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.174894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:21:06.177419 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.177386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1340cfc-2d2c-4048-af47-f70bc4fa1bb2-metrics-certs\") pod \"network-metrics-daemon-8dm9k\" (UID: \"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2\") " pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:21:06.238211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.238183 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2jpl6\"" Jul 02 20:21:06.246350 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.246328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8dm9k" Jul 02 20:21:06.374373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.374337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8dm9k"] Jul 02 20:21:06.377352 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:21:06.377326 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1340cfc_2d2c_4048_af47_f70bc4fa1bb2.slice/crio-71b3bda12b27527416a8176690b720b5fc20fe7d39a5e803fcef138399632f5b WatchSource:0}: Error finding container 71b3bda12b27527416a8176690b720b5fc20fe7d39a5e803fcef138399632f5b: Status 404 returned error can't find the container with id 71b3bda12b27527416a8176690b720b5fc20fe7d39a5e803fcef138399632f5b Jul 02 20:21:06.379644 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.379629 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:21:06.616422 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:06.616388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8dm9k" event={"ID":"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2","Type":"ContainerStarted","Data":"71b3bda12b27527416a8176690b720b5fc20fe7d39a5e803fcef138399632f5b"} Jul 02 20:21:07.621483 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:07.621447 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8dm9k" event={"ID":"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2","Type":"ContainerStarted","Data":"10cda0715e5b976bb061315506616152ac83b85709783b5624a46887bc9d6e15"} Jul 02 20:21:07.621870 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:07.621514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8dm9k" event={"ID":"c1340cfc-2d2c-4048-af47-f70bc4fa1bb2","Type":"ContainerStarted","Data":"8f1f7039f317d74a51f6d7294c9ffd5e863ad4000cbe963191afbafb1e6f08ce"} Jul 02 20:21:07.636199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:07.636147 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8dm9k" podStartSLOduration=374.692578192 podStartE2EDuration="6m15.636130806s" totalImagesPullingTime="943.552614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:21:07.634773479 +0000 UTC m=+375.840160107" watchObservedRunningTime="2026-07-02 20:21:07.636130806 +0000 UTC m=+375.841517453" Jul 02 20:21:58.948658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:58.948624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:21:58.964789 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:58.964762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:21:59.827535 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:21:59.827509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 20:24:24.392052 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.392018 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n"] Jul 02 20:24:24.395274 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.395257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.399551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.399530 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 20:24:24.399551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.399552 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xncpr\"" Jul 02 20:24:24.399795 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.399593 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 20:24:24.399795 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.399685 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 20:24:24.399947 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.399812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 20:24:24.413228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.413207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n"] Jul 02 20:24:24.542066 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.542033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.542270 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.542081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.542270 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.542112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsgh\" (UniqueName: \"kubernetes.io/projected/11751f38-8ff4-4712-9f3a-36f70e815271-kube-api-access-hxsgh\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.542270 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.542151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.542270 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.542206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.643675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.643566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.643675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.643626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.643675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.643660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.643933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.643680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsgh\" (UniqueName: \"kubernetes.io/projected/11751f38-8ff4-4712-9f3a-36f70e815271-kube-api-access-hxsgh\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.643933 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.643700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.644204 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.644177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.646125 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.646098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11751f38-8ff4-4712-9f3a-36f70e815271-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.646328 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.646307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.646453 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.646434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11751f38-8ff4-4712-9f3a-36f70e815271-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.653555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.653534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsgh\" (UniqueName: \"kubernetes.io/projected/11751f38-8ff4-4712-9f3a-36f70e815271-kube-api-access-hxsgh\") pod \"opendatahub-operator-controller-manager-78b478dcdf-ksc2n\" (UID: \"11751f38-8ff4-4712-9f3a-36f70e815271\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.705565 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.705528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:24.870122 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:24.870097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n"] Jul 02 20:24:24.872349 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:24:24.872324 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11751f38_8ff4_4712_9f3a_36f70e815271.slice/crio-4c9f82c088ac77d3fcf8d5ead1850823b5cd581846091227011865dced41fea0 WatchSource:0}: Error finding container 4c9f82c088ac77d3fcf8d5ead1850823b5cd581846091227011865dced41fea0: Status 404 returned error can't find the container with id 4c9f82c088ac77d3fcf8d5ead1850823b5cd581846091227011865dced41fea0 Jul 02 20:24:25.318717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:25.318684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" event={"ID":"11751f38-8ff4-4712-9f3a-36f70e815271","Type":"ContainerStarted","Data":"4c9f82c088ac77d3fcf8d5ead1850823b5cd581846091227011865dced41fea0"} Jul 02 20:24:28.331858 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:28.331802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" event={"ID":"11751f38-8ff4-4712-9f3a-36f70e815271","Type":"ContainerStarted","Data":"6726f3f862741dc485cb4e12f9bd49f67fd05f733bc2de0ee7db9bddb33b82c9"} Jul 02 20:24:29.336759 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:29.336668 2579 generic.go:356] "Generic (PLEG): container finished" podID="11751f38-8ff4-4712-9f3a-36f70e815271" containerID="6726f3f862741dc485cb4e12f9bd49f67fd05f733bc2de0ee7db9bddb33b82c9" exitCode=0 Jul 02 20:24:29.337151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:29.336762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" event={"ID":"11751f38-8ff4-4712-9f3a-36f70e815271","Type":"ContainerDied","Data":"6726f3f862741dc485cb4e12f9bd49f67fd05f733bc2de0ee7db9bddb33b82c9"} Jul 02 20:24:30.341321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:30.341289 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" event={"ID":"11751f38-8ff4-4712-9f3a-36f70e815271","Type":"ContainerStarted","Data":"0054c2215ae60f7185b3ae00afad1264687beab214fbbe4eb3c2bf102f492066"} Jul 02 20:24:30.341694 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:30.341426 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:30.365685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:30.365633 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" podStartSLOduration=1.938371391 podStartE2EDuration="6.365612183s" totalImagesPullingTime="3.378878659s" totalInitContainerRuntime="1.048362133s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:30.36403295 +0000 UTC m=+578.569419591" watchObservedRunningTime="2026-07-02 20:24:30.365612183 +0000 UTC m=+578.570998820" Jul 02 20:24:41.348228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:41.348194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-ksc2n" Jul 02 20:24:49.431866 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.431818 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-fd64s"] Jul 02 20:24:49.435073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.435057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.445152 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.445119 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 20:24:49.446233 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.446214 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 20:24:49.446233 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.446226 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4bt7t\"" Jul 02 20:24:49.446401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.446214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 20:24:49.446401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.446252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 20:24:49.460923 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.460897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-fd64s"] Jul 02 20:24:49.536177 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.536123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27691e00-296f-40e2-9c6e-09bd198c10f3-tls-certs\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.536373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.536215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27691e00-296f-40e2-9c6e-09bd198c10f3-tmp\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.536373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.536237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjht\" (UniqueName: \"kubernetes.io/projected/27691e00-296f-40e2-9c6e-09bd198c10f3-kube-api-access-znjht\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.542091 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.542060 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4mk8s"] Jul 02 20:24:49.545422 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.545392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.548535 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.548509 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q46bh\"" Jul 02 20:24:49.548765 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.548746 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 02 20:24:49.563985 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.563959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4mk8s"] Jul 02 20:24:49.637493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.637455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.637493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.637500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27691e00-296f-40e2-9c6e-09bd198c10f3-tmp\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.637706 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.637520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjht\" (UniqueName: \"kubernetes.io/projected/27691e00-296f-40e2-9c6e-09bd198c10f3-kube-api-access-znjht\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.637706 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.637616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92mnh\" (UniqueName: \"kubernetes.io/projected/c35bd262-954b-4fa0-9981-dccf2efedd71-kube-api-access-92mnh\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.637706 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.637656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27691e00-296f-40e2-9c6e-09bd198c10f3-tls-certs\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.639888 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.639858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27691e00-296f-40e2-9c6e-09bd198c10f3-tmp\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.640164 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.640145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27691e00-296f-40e2-9c6e-09bd198c10f3-tls-certs\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.648157 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.648124 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjht\" (UniqueName: \"kubernetes.io/projected/27691e00-296f-40e2-9c6e-09bd198c10f3-kube-api-access-znjht\") pod \"kube-auth-proxy-d957c959b-fd64s\" (UID: \"27691e00-296f-40e2-9c6e-09bd198c10f3\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.738607 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.738507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92mnh\" (UniqueName: \"kubernetes.io/projected/c35bd262-954b-4fa0-9981-dccf2efedd71-kube-api-access-92mnh\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.738607 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.738575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.738780 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:24:49.738694 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 02 20:24:49.738780 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:24:49.738772 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert podName:c35bd262-954b-4fa0-9981-dccf2efedd71 nodeName:}" failed. No retries permitted until 2026-07-02 20:24:50.238750457 +0000 UTC m=+598.444137065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert") pod "odh-model-controller-7998c67dc4-4mk8s" (UID: "c35bd262-954b-4fa0-9981-dccf2efedd71") : secret "odh-model-controller-webhook-cert" not found Jul 02 20:24:49.743932 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.743906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" Jul 02 20:24:49.748137 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.748114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92mnh\" (UniqueName: \"kubernetes.io/projected/c35bd262-954b-4fa0-9981-dccf2efedd71-kube-api-access-92mnh\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:49.884495 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:49.884463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-fd64s"] Jul 02 20:24:49.889184 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:24:49.889156 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27691e00_296f_40e2_9c6e_09bd198c10f3.slice/crio-5319efe5c2c9f17ff7ec6266715967404f63b10eaec78f0bccbf0af3e0fcaf43 WatchSource:0}: Error finding container 5319efe5c2c9f17ff7ec6266715967404f63b10eaec78f0bccbf0af3e0fcaf43: Status 404 returned error can't find the container with id 5319efe5c2c9f17ff7ec6266715967404f63b10eaec78f0bccbf0af3e0fcaf43 Jul 02 20:24:50.243862 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.243804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:50.246670 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.246644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35bd262-954b-4fa0-9981-dccf2efedd71-cert\") pod \"odh-model-controller-7998c67dc4-4mk8s\" (UID: \"c35bd262-954b-4fa0-9981-dccf2efedd71\") " pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:50.413043 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.413008 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" event={"ID":"27691e00-296f-40e2-9c6e-09bd198c10f3","Type":"ContainerStarted","Data":"5319efe5c2c9f17ff7ec6266715967404f63b10eaec78f0bccbf0af3e0fcaf43"} Jul 02 20:24:50.438680 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.438648 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckzps"] Jul 02 20:24:50.443002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.442981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.445526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.445501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-c4c4c\"" Jul 02 20:24:50.445805 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.445792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 02 20:24:50.453607 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.453577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckzps"] Jul 02 20:24:50.455925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.455902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:50.546465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.546432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.546621 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.546482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klm4t\" (UniqueName: \"kubernetes.io/projected/fc4ea332-04e9-4127-9483-8196f79f50e7-kube-api-access-klm4t\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.611527 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.611486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4mk8s"] Jul 02 20:24:50.615537 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:24:50.615507 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35bd262_954b_4fa0_9981_dccf2efedd71.slice/crio-424f9d4c61c92f7e5a20478cb58614b89730ab78541f044b0048d957071eb88f WatchSource:0}: Error finding container 424f9d4c61c92f7e5a20478cb58614b89730ab78541f044b0048d957071eb88f: Status 404 returned error can't find the container with id 424f9d4c61c92f7e5a20478cb58614b89730ab78541f044b0048d957071eb88f Jul 02 20:24:50.647494 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.647459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.647671 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.647511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klm4t\" (UniqueName: \"kubernetes.io/projected/fc4ea332-04e9-4127-9483-8196f79f50e7-kube-api-access-klm4t\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.647946 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:24:50.647921 2579 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 02 20:24:50.648055 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:24:50.648014 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs podName:fc4ea332-04e9-4127-9483-8196f79f50e7 nodeName:}" failed. No retries permitted until 2026-07-02 20:24:51.147991747 +0000 UTC m=+599.353378356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs") pod "model-serving-api-54d89bd5fc-ckzps" (UID: "fc4ea332-04e9-4127-9483-8196f79f50e7") : secret "model-serving-api-tls" not found Jul 02 20:24:50.660197 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.660137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klm4t\" (UniqueName: \"kubernetes.io/projected/fc4ea332-04e9-4127-9483-8196f79f50e7-kube-api-access-klm4t\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:50.775161 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.775068 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:24:50.778879 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.778826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.781885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.781861 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 20:24:50.781885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.781877 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 20:24:50.782668 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.782618 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 20:24:50.782668 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.782651 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wqgx8\"" Jul 02 20:24:50.782808 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.782684 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 20:24:50.805898 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.805869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:24:50.850222 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm5t\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.850577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.850487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951064 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm5t\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.951509 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.951296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.952143 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.952114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.954316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.954262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.954515 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.954495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.954591 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.954570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.955321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.955301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.961345 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.961315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:50.961961 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:50.961927 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm5t\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t\") pod \"istiod-openshift-gateway-7574459656-kvzvn\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:51.091425 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.091337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:51.152831 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.152745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:51.156271 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.156242 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc4ea332-04e9-4127-9483-8196f79f50e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-ckzps\" (UID: \"fc4ea332-04e9-4127-9483-8196f79f50e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:51.239984 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.239956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:24:51.241600 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:24:51.241576 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d08aee_117d_4877_a734_be2dea7218d5.slice/crio-0bf3e451f401320a080d2035ed70082bd84712a690db9f17e8d0cc7cd46adc96 WatchSource:0}: Error finding container 0bf3e451f401320a080d2035ed70082bd84712a690db9f17e8d0cc7cd46adc96: Status 404 returned error can't find the container with id 0bf3e451f401320a080d2035ed70082bd84712a690db9f17e8d0cc7cd46adc96 Jul 02 20:24:51.357029 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.356935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:51.420658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.420620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" event={"ID":"e9d08aee-117d-4877-a734-be2dea7218d5","Type":"ContainerStarted","Data":"0bf3e451f401320a080d2035ed70082bd84712a690db9f17e8d0cc7cd46adc96"} Jul 02 20:24:51.422247 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.422219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" event={"ID":"c35bd262-954b-4fa0-9981-dccf2efedd71","Type":"ContainerStarted","Data":"424f9d4c61c92f7e5a20478cb58614b89730ab78541f044b0048d957071eb88f"} Jul 02 20:24:51.512464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:51.512431 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ckzps"] Jul 02 20:24:51.515815 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:24:51.515728 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ea332_04e9_4127_9483_8196f79f50e7.slice/crio-66d573cc185fa41575244ca819ba79dc04207f1d3ec854d1d838aacb89ffa261 WatchSource:0}: Error finding container 66d573cc185fa41575244ca819ba79dc04207f1d3ec854d1d838aacb89ffa261: Status 404 returned error can't find the container with id 66d573cc185fa41575244ca819ba79dc04207f1d3ec854d1d838aacb89ffa261 Jul 02 20:24:52.430813 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:52.430773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" event={"ID":"fc4ea332-04e9-4127-9483-8196f79f50e7","Type":"ContainerStarted","Data":"66d573cc185fa41575244ca819ba79dc04207f1d3ec854d1d838aacb89ffa261"} Jul 02 20:24:56.055999 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.055967 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 20:24:56.061905 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.061864 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:24:56.062019 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.061949 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:24:56.083712 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.083527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:24:56.087743 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.087670 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:24:56.448855 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.448810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" event={"ID":"e9d08aee-117d-4877-a734-be2dea7218d5","Type":"ContainerStarted","Data":"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b"} Jul 02 20:24:56.449716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.449639 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:24:56.451797 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.451774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" event={"ID":"c35bd262-954b-4fa0-9981-dccf2efedd71","Type":"ContainerStarted","Data":"f35dcda78c8dc8f0f9f437cd6f146e9227d2b96564dcb4ce85a242b8df345bdb"} Jul 02 20:24:56.452277 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.452260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:24:56.454156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.454133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" event={"ID":"27691e00-296f-40e2-9c6e-09bd198c10f3","Type":"ContainerStarted","Data":"36a130ba00fbf0dbcde5bf9a9148014b3980a1277590a9a29775b4984217266d"} Jul 02 20:24:56.456642 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.456327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" event={"ID":"fc4ea332-04e9-4127-9483-8196f79f50e7","Type":"ContainerStarted","Data":"f5ac6a2d12dc82e983ac4d2fb71144b2ac8981032bb9543e3a05b4811f0d8ea2"} Jul 02 20:24:56.456642 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.456406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:24:56.475289 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.475228 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" podStartSLOduration=1.6573399549999999 podStartE2EDuration="6.475209416s" totalImagesPullingTime="4.817869461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:56.472332925 +0000 UTC m=+604.677719595" watchObservedRunningTime="2026-07-02 20:24:56.475209416 +0000 UTC m=+604.680596046" Jul 02 20:24:56.490704 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.490640 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" podStartSLOduration=2.037054915 podStartE2EDuration="7.490618488s" totalImagesPullingTime="5.453563573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:56.489060887 +0000 UTC m=+604.694447542" watchObservedRunningTime="2026-07-02 20:24:56.490618488 +0000 UTC m=+604.696005118" Jul 02 20:24:56.509074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.509021 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d957c959b-fd64s" podStartSLOduration=1.346907953 podStartE2EDuration="7.509001862s" totalImagesPullingTime="6.162093909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:56.508544222 +0000 UTC m=+604.713930872" watchObservedRunningTime="2026-07-02 20:24:56.509001862 +0000 UTC m=+604.714388492" Jul 02 20:24:56.541638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:56.535064 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" podStartSLOduration=1.995885356 podStartE2EDuration="6.535044427s" totalImagesPullingTime="4.539159071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:56.534429318 +0000 UTC m=+604.739815972" watchObservedRunningTime="2026-07-02 20:24:56.535044427 +0000 UTC m=+604.740431056" Jul 02 20:24:57.462372 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:57.462328 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kvzvn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 20:24:57.462723 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:57.462403 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 20:24:58.464993 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:24:58.464968 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:25:01.568530 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.568495 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7b44"] Jul 02 20:25:01.571482 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.571466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.577390 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.577362 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-b2g9l\"" Jul 02 20:25:01.577390 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.577371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 02 20:25:01.599316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.599284 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph"] Jul 02 20:25:01.602310 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.602290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.606396 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.606377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fzg94\"" Jul 02 20:25:01.607226 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.607201 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 02 20:25:01.607389 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.607359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7b44"] Jul 02 20:25:01.634784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.634746 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph"] Jul 02 20:25:01.650273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.650245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np64k\" (UniqueName: \"kubernetes.io/projected/d4056c24-33f2-48d9-813f-5b001e4b9e35-kube-api-access-np64k\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.650430 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.650293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbm9p\" (UniqueName: \"kubernetes.io/projected/f41800cf-ceac-4609-a972-d459ea3ea166-kube-api-access-tbm9p\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.650430 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.650317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.650430 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.650393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4056c24-33f2-48d9-813f-5b001e4b9e35-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.751456 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.751413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbm9p\" (UniqueName: \"kubernetes.io/projected/f41800cf-ceac-4609-a972-d459ea3ea166-kube-api-access-tbm9p\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.751677 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:01.751593 2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 02 20:25:01.751739 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:01.751691 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert podName:f41800cf-ceac-4609-a972-d459ea3ea166 nodeName:}" failed. No retries permitted until 2026-07-02 20:25:02.251666764 +0000 UTC m=+610.457053374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert") pod "llmisvc-controller-manager-777c6bd7d5-th9ph" (UID: "f41800cf-ceac-4609-a972-d459ea3ea166") : secret "llmisvc-webhook-server-cert" not found Jul 02 20:25:01.752155 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.752125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.752373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.752347 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4056c24-33f2-48d9-813f-5b001e4b9e35-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.752570 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.752556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np64k\" (UniqueName: \"kubernetes.io/projected/d4056c24-33f2-48d9-813f-5b001e4b9e35-kube-api-access-np64k\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.756245 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.756212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4056c24-33f2-48d9-813f-5b001e4b9e35-cert\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.768332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.768304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbm9p\" (UniqueName: \"kubernetes.io/projected/f41800cf-ceac-4609-a972-d459ea3ea166-kube-api-access-tbm9p\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:01.771107 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.771078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np64k\" (UniqueName: \"kubernetes.io/projected/d4056c24-33f2-48d9-813f-5b001e4b9e35-kube-api-access-np64k\") pod \"kserve-controller-manager-d6f4b65b8-r7b44\" (UID: \"d4056c24-33f2-48d9-813f-5b001e4b9e35\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:01.881799 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:01.881769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:02.048829 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:02.048744 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r7b44"] Jul 02 20:25:02.051443 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:25:02.051414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4056c24_33f2_48d9_813f_5b001e4b9e35.slice/crio-03cfe37f0aefb9c528d67345ee35da871118f921b594e8886af04b901b09ec75 WatchSource:0}: Error finding container 03cfe37f0aefb9c528d67345ee35da871118f921b594e8886af04b901b09ec75: Status 404 returned error can't find the container with id 03cfe37f0aefb9c528d67345ee35da871118f921b594e8886af04b901b09ec75 Jul 02 20:25:02.258294 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:02.258189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:02.258448 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:02.258351 2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 02 20:25:02.258448 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:02.258434 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert podName:f41800cf-ceac-4609-a972-d459ea3ea166 nodeName:}" failed. No retries permitted until 2026-07-02 20:25:03.258417837 +0000 UTC m=+611.463804444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert") pod "llmisvc-controller-manager-777c6bd7d5-th9ph" (UID: "f41800cf-ceac-4609-a972-d459ea3ea166") : secret "llmisvc-webhook-server-cert" not found Jul 02 20:25:02.480787 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:02.480755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" event={"ID":"d4056c24-33f2-48d9-813f-5b001e4b9e35","Type":"ContainerStarted","Data":"03cfe37f0aefb9c528d67345ee35da871118f921b594e8886af04b901b09ec75"} Jul 02 20:25:02.862811 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:02.862757 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:25:02.863264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:02.863046 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" containerName="discovery" containerID="cri-o://6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b" gracePeriod=30 Jul 02 20:25:03.117802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.117744 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:25:03.266777 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.266746 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccm5t\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.266777 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.266779 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.266802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.266949 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267004 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267208 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267051 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267208 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts\") pod \"e9d08aee-117d-4877-a734-be2dea7218d5\" (UID: \"e9d08aee-117d-4877-a734-be2dea7218d5\") " Jul 02 20:25:03.267312 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267223 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:25:03.267540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:03.267540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.267434 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-ca-configmap\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.270059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270000 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t" (OuterVolumeSpecName: "kube-api-access-ccm5t") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "kube-api-access-ccm5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:25:03.270059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270000 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:25:03.270269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270090 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs" (OuterVolumeSpecName: "local-certs") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:25:03.270269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270112 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:25:03.270269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270184 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts" (OuterVolumeSpecName: "cacerts") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:25:03.270822 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270806 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e9d08aee-117d-4877-a734-be2dea7218d5" (UID: "e9d08aee-117d-4877-a734-be2dea7218d5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:25:03.270922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.270887 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f41800cf-ceac-4609-a972-d459ea3ea166-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th9ph\" (UID: \"f41800cf-ceac-4609-a972-d459ea3ea166\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368793 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-kubeconfig\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368867 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9d08aee-117d-4877-a734-be2dea7218d5-local-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368881 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-cacerts\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368892 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccm5t\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-kube-api-access-ccm5t\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368906 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9d08aee-117d-4877-a734-be2dea7218d5-istio-token\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.368925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.368917 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9d08aee-117d-4877-a734-be2dea7218d5-istio-csr-dns-cert\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:03.413531 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.413494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:03.486457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.486419 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9d08aee-117d-4877-a734-be2dea7218d5" containerID="6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b" exitCode=0 Jul 02 20:25:03.486457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.486466 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" Jul 02 20:25:03.486675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.486510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" event={"ID":"e9d08aee-117d-4877-a734-be2dea7218d5","Type":"ContainerDied","Data":"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b"} Jul 02 20:25:03.486675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.486577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn" event={"ID":"e9d08aee-117d-4877-a734-be2dea7218d5","Type":"ContainerDied","Data":"0bf3e451f401320a080d2035ed70082bd84712a690db9f17e8d0cc7cd46adc96"} Jul 02 20:25:03.486675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.486602 2579 scope.go:117] "RemoveContainer" containerID="6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b" Jul 02 20:25:03.495874 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.495829 2579 scope.go:117] "RemoveContainer" containerID="6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b" Jul 02 20:25:03.496173 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:03.496148 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b\": container with ID starting with 6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b not found: ID does not exist" containerID="6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b" Jul 02 20:25:03.496273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.496180 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b"} err="failed to get container status \"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b\": rpc error: code = NotFound desc = could not find container \"6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b\": container with ID starting with 6abb73d237cbec3ec855d50182c584ab82d375dcd4ba72ca006456985e52fb6b not found: ID does not exist" Jul 02 20:25:03.521037 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.521009 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:25:03.524609 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.524580 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kvzvn"] Jul 02 20:25:03.559514 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:03.559484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph"] Jul 02 20:25:03.563225 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:25:03.563196 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf41800cf_ceac_4609_a972_d459ea3ea166.slice/crio-dc3cb220fd82df15c9e66753295130e52c450249d805bfc96d7b0175805acb8a WatchSource:0}: Error finding container dc3cb220fd82df15c9e66753295130e52c450249d805bfc96d7b0175805acb8a: Status 404 returned error can't find the container with id dc3cb220fd82df15c9e66753295130e52c450249d805bfc96d7b0175805acb8a Jul 02 20:25:04.340675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:04.340624 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" path="/var/lib/kubelet/pods/e9d08aee-117d-4877-a734-be2dea7218d5/volumes" Jul 02 20:25:04.494355 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:04.494320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" event={"ID":"f41800cf-ceac-4609-a972-d459ea3ea166","Type":"ContainerStarted","Data":"dc3cb220fd82df15c9e66753295130e52c450249d805bfc96d7b0175805acb8a"} Jul 02 20:25:06.504677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:06.504640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" event={"ID":"d4056c24-33f2-48d9-813f-5b001e4b9e35","Type":"ContainerStarted","Data":"370a5d26915f061c172b87bedcb4d3b69e8969e0c5395daf4214e4629ad8e239"} Jul 02 20:25:06.505086 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:06.504762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:06.521772 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:06.521712 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" podStartSLOduration=2.464639525 podStartE2EDuration="5.521696132s" totalImagesPullingTime="3.057056607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:25:06.521092783 +0000 UTC m=+614.726479425" watchObservedRunningTime="2026-07-02 20:25:06.521696132 +0000 UTC m=+614.727082761" Jul 02 20:25:07.464874 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:07.464832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ckzps" Jul 02 20:25:07.513126 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:07.513036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" event={"ID":"f41800cf-ceac-4609-a972-d459ea3ea166","Type":"ContainerStarted","Data":"852803a25845bc81c9f81df109f8f0947e57fec19ac1d9b440e0a74bfe4eb513"} Jul 02 20:25:07.513562 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:07.513123 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:07.530742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:07.530685 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" podStartSLOduration=2.851141416 podStartE2EDuration="6.530667844s" totalImagesPullingTime="3.679526428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:25:07.530463984 +0000 UTC m=+615.735850641" watchObservedRunningTime="2026-07-02 20:25:07.530667844 +0000 UTC m=+615.736054472" Jul 02 20:25:08.466387 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:08.466352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4mk8s" Jul 02 20:25:37.518265 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:37.518226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r7b44" Jul 02 20:25:38.520236 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:38.520203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th9ph" Jul 02 20:25:45.080316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.080276 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:45.080701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.080634 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" containerName="discovery" Jul 02 20:25:45.080701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.080652 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" containerName="discovery" Jul 02 20:25:45.080769 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.080716 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d08aee-117d-4877-a734-be2dea7218d5" containerName="discovery" Jul 02 20:25:45.083727 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.083710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:45.086856 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.086819 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7j8zh\"" Jul 02 20:25:45.086985 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.086822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 20:25:45.087966 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.087945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 20:25:45.094410 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.094381 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:45.139405 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.139375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv652\" (UniqueName: \"kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652\") pod \"kuadrant-operator-catalog-w7gww\" (UID: \"f0df79e9-7c67-45fa-89dc-29620660f5a1\") " pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:45.240545 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.240508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv652\" (UniqueName: \"kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652\") pod \"kuadrant-operator-catalog-w7gww\" (UID: \"f0df79e9-7c67-45fa-89dc-29620660f5a1\") " pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:45.249629 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.249593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv652\" (UniqueName: \"kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652\") pod \"kuadrant-operator-catalog-w7gww\" (UID: \"f0df79e9-7c67-45fa-89dc-29620660f5a1\") " pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:45.395543 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.395501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:45.437138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.437105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:45.527855 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.527818 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:45.532080 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:25:45.532055 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0df79e9_7c67_45fa_89dc_29620660f5a1.slice/crio-a644c33cc522c28c743e7f3390f25cd8e671d16a33577b04f4e8ec3579d97bf0 WatchSource:0}: Error finding container a644c33cc522c28c743e7f3390f25cd8e671d16a33577b04f4e8ec3579d97bf0: Status 404 returned error can't find the container with id a644c33cc522c28c743e7f3390f25cd8e671d16a33577b04f4e8ec3579d97bf0 Jul 02 20:25:45.652335 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.652247 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wc5gs"] Jul 02 20:25:45.656659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.656634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:45.663588 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.663512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wc5gs"] Jul 02 20:25:45.693004 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.692968 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" event={"ID":"f0df79e9-7c67-45fa-89dc-29620660f5a1","Type":"ContainerStarted","Data":"a644c33cc522c28c743e7f3390f25cd8e671d16a33577b04f4e8ec3579d97bf0"} Jul 02 20:25:45.745093 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.745057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ch7\" (UniqueName: \"kubernetes.io/projected/a962f7e1-3278-45bc-b5c4-a26b54d24f00-kube-api-access-z4ch7\") pod \"kuadrant-operator-catalog-wc5gs\" (UID: \"a962f7e1-3278-45bc-b5c4-a26b54d24f00\") " pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:45.845637 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.845597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ch7\" (UniqueName: \"kubernetes.io/projected/a962f7e1-3278-45bc-b5c4-a26b54d24f00-kube-api-access-z4ch7\") pod \"kuadrant-operator-catalog-wc5gs\" (UID: \"a962f7e1-3278-45bc-b5c4-a26b54d24f00\") " pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:45.854466 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.854433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ch7\" (UniqueName: \"kubernetes.io/projected/a962f7e1-3278-45bc-b5c4-a26b54d24f00-kube-api-access-z4ch7\") pod \"kuadrant-operator-catalog-wc5gs\" (UID: \"a962f7e1-3278-45bc-b5c4-a26b54d24f00\") " pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:45.967070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:45.966994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:46.149194 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:46.149106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wc5gs"] Jul 02 20:25:46.217014 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:25:46.216972 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda962f7e1_3278_45bc_b5c4_a26b54d24f00.slice/crio-25d99ec2197a271160c22c446d1dd6cf703e9b6abb3955fe580e8d58762ed542 WatchSource:0}: Error finding container 25d99ec2197a271160c22c446d1dd6cf703e9b6abb3955fe580e8d58762ed542: Status 404 returned error can't find the container with id 25d99ec2197a271160c22c446d1dd6cf703e9b6abb3955fe580e8d58762ed542 Jul 02 20:25:46.699900 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:46.699866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" event={"ID":"a962f7e1-3278-45bc-b5c4-a26b54d24f00","Type":"ContainerStarted","Data":"25d99ec2197a271160c22c446d1dd6cf703e9b6abb3955fe580e8d58762ed542"} Jul 02 20:25:47.708482 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:47.708445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" event={"ID":"a962f7e1-3278-45bc-b5c4-a26b54d24f00","Type":"ContainerStarted","Data":"e14a9fce1b8ce6407325b56b23682d8b9a7c49ad92932384ecde64c9845b024e"} Jul 02 20:25:47.710620 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:47.710576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" event={"ID":"f0df79e9-7c67-45fa-89dc-29620660f5a1","Type":"ContainerStarted","Data":"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82"} Jul 02 20:25:47.710749 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:47.710672 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" podUID="f0df79e9-7c67-45fa-89dc-29620660f5a1" containerName="registry-server" containerID="cri-o://c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82" gracePeriod=2 Jul 02 20:25:47.727314 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:47.727263 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" podStartSLOduration=1.3036705610000001 podStartE2EDuration="2.727244687s" totalImagesPullingTime="1.423574126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:25:47.726392942 +0000 UTC m=+655.931779583" watchObservedRunningTime="2026-07-02 20:25:47.727244687 +0000 UTC m=+655.932631316" Jul 02 20:25:47.745917 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:47.745830 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" podStartSLOduration=0.67878746 podStartE2EDuration="2.745807518s" totalImagesPullingTime="2.067020058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:25:47.744942303 +0000 UTC m=+655.950328933" watchObservedRunningTime="2026-07-02 20:25:47.745807518 +0000 UTC m=+655.951194149" Jul 02 20:25:48.052287 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.052265 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:48.166343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.166301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv652\" (UniqueName: \"kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652\") pod \"f0df79e9-7c67-45fa-89dc-29620660f5a1\" (UID: \"f0df79e9-7c67-45fa-89dc-29620660f5a1\") " Jul 02 20:25:48.168606 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.168572 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652" (OuterVolumeSpecName: "kube-api-access-bv652") pod "f0df79e9-7c67-45fa-89dc-29620660f5a1" (UID: "f0df79e9-7c67-45fa-89dc-29620660f5a1"). InnerVolumeSpecName "kube-api-access-bv652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:25:48.267675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.267584 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv652\" (UniqueName: \"kubernetes.io/projected/f0df79e9-7c67-45fa-89dc-29620660f5a1-kube-api-access-bv652\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:25:48.715717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.715677 2579 generic.go:356] "Generic (PLEG): container finished" podID="f0df79e9-7c67-45fa-89dc-29620660f5a1" containerID="c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82" exitCode=0 Jul 02 20:25:48.716134 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.715740 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" Jul 02 20:25:48.716134 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.715768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" event={"ID":"f0df79e9-7c67-45fa-89dc-29620660f5a1","Type":"ContainerDied","Data":"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82"} Jul 02 20:25:48.716134 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.715818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-w7gww" event={"ID":"f0df79e9-7c67-45fa-89dc-29620660f5a1","Type":"ContainerDied","Data":"a644c33cc522c28c743e7f3390f25cd8e671d16a33577b04f4e8ec3579d97bf0"} Jul 02 20:25:48.716134 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.715872 2579 scope.go:117] "RemoveContainer" containerID="c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82" Jul 02 20:25:48.724504 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.724480 2579 scope.go:117] "RemoveContainer" containerID="c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82" Jul 02 20:25:48.724789 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:25:48.724765 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82\": container with ID starting with c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82 not found: ID does not exist" containerID="c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82" Jul 02 20:25:48.724906 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.724800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82"} err="failed to get container status \"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82\": rpc error: code = NotFound desc = could not find container \"c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82\": container with ID starting with c28f3ea473e28807ff961819a35dc2d7466f209333d8c8743121e5c15e667a82 not found: ID does not exist" Jul 02 20:25:48.733968 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.733940 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:48.738258 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:48.738234 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-w7gww"] Jul 02 20:25:50.339401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:50.339368 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0df79e9-7c67-45fa-89dc-29620660f5a1" path="/var/lib/kubelet/pods/f0df79e9-7c67-45fa-89dc-29620660f5a1/volumes" Jul 02 20:25:55.967460 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:55.967413 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:55.968002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:55.967521 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:55.990625 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:55.990594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:25:56.778685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:25:56.778657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wc5gs" Jul 02 20:26:19.132509 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.132471 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w"] Jul 02 20:26:19.133077 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.132819 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0df79e9-7c67-45fa-89dc-29620660f5a1" containerName="registry-server" Jul 02 20:26:19.133077 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.132833 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0df79e9-7c67-45fa-89dc-29620660f5a1" containerName="registry-server" Jul 02 20:26:19.133077 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.132917 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0df79e9-7c67-45fa-89dc-29620660f5a1" containerName="registry-server" Jul 02 20:26:19.139453 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.139432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.144582 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.144560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6v7zk\"" Jul 02 20:26:19.167133 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.167103 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w"] Jul 02 20:26:19.232932 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.232892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.233104 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.233015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.334273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.334229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.334459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.334301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.334719 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.334700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.349398 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.349371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.449127 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.449037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:19.595874 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.595849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w"] Jul 02 20:26:19.598557 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:19.598529 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff30e57_f092_4ee5_ac9e_9113236b724a.slice/crio-405210e328eab1acffe0a7cd787304b533b4dbe54a93889fc6ce9c3e6cbe3a02 WatchSource:0}: Error finding container 405210e328eab1acffe0a7cd787304b533b4dbe54a93889fc6ce9c3e6cbe3a02: Status 404 returned error can't find the container with id 405210e328eab1acffe0a7cd787304b533b4dbe54a93889fc6ce9c3e6cbe3a02 Jul 02 20:26:19.600978 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.600962 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:26:19.852868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:19.852762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" event={"ID":"0ff30e57-f092-4ee5-ac9e-9113236b724a","Type":"ContainerStarted","Data":"405210e328eab1acffe0a7cd787304b533b4dbe54a93889fc6ce9c3e6cbe3a02"} Jul 02 20:26:21.489247 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.489211 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv"] Jul 02 20:26:21.492568 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.492551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:21.496059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.496038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kbtml\"" Jul 02 20:26:21.509027 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.508998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv"] Jul 02 20:26:21.555315 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.555281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftnf\" (UniqueName: \"kubernetes.io/projected/5bde44fc-d3a3-4236-bcc6-eea5acf608c6-kube-api-access-4ftnf\") pod \"authorino-operator-75cc8d5cb7-pzxrv\" (UID: \"5bde44fc-d3a3-4236-bcc6-eea5acf608c6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:21.656352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.656310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftnf\" (UniqueName: \"kubernetes.io/projected/5bde44fc-d3a3-4236-bcc6-eea5acf608c6-kube-api-access-4ftnf\") pod \"authorino-operator-75cc8d5cb7-pzxrv\" (UID: \"5bde44fc-d3a3-4236-bcc6-eea5acf608c6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:21.675515 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.675479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftnf\" (UniqueName: \"kubernetes.io/projected/5bde44fc-d3a3-4236-bcc6-eea5acf608c6-kube-api-access-4ftnf\") pod \"authorino-operator-75cc8d5cb7-pzxrv\" (UID: \"5bde44fc-d3a3-4236-bcc6-eea5acf608c6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:21.803030 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.802935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:21.960250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:21.960216 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv"] Jul 02 20:26:21.963915 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:21.963879 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde44fc_d3a3_4236_bcc6_eea5acf608c6.slice/crio-ca7fda54572512d763bbe0548d0046e4e061fd800b75e5945d9437c4967b7642 WatchSource:0}: Error finding container ca7fda54572512d763bbe0548d0046e4e061fd800b75e5945d9437c4967b7642: Status 404 returned error can't find the container with id ca7fda54572512d763bbe0548d0046e4e061fd800b75e5945d9437c4967b7642 Jul 02 20:26:22.867007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:22.866969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" event={"ID":"5bde44fc-d3a3-4236-bcc6-eea5acf608c6","Type":"ContainerStarted","Data":"ca7fda54572512d763bbe0548d0046e4e061fd800b75e5945d9437c4967b7642"} Jul 02 20:26:23.510441 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.510320 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq"] Jul 02 20:26:23.514358 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.514330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:23.519649 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.519436 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 02 20:26:23.520240 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.520218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7hfmz\"" Jul 02 20:26:23.545642 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.545608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq"] Jul 02 20:26:23.573601 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.573531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcgb\" (UniqueName: \"kubernetes.io/projected/da68f9e7-997a-4697-96c7-c6983f10a42a-kube-api-access-9lcgb\") pod \"dns-operator-controller-manager-864dd78c4c-qskkq\" (UID: \"da68f9e7-997a-4697-96c7-c6983f10a42a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:23.674894 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.674828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcgb\" (UniqueName: \"kubernetes.io/projected/da68f9e7-997a-4697-96c7-c6983f10a42a-kube-api-access-9lcgb\") pod \"dns-operator-controller-manager-864dd78c4c-qskkq\" (UID: \"da68f9e7-997a-4697-96c7-c6983f10a42a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:23.694822 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.694758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcgb\" (UniqueName: \"kubernetes.io/projected/da68f9e7-997a-4697-96c7-c6983f10a42a-kube-api-access-9lcgb\") pod \"dns-operator-controller-manager-864dd78c4c-qskkq\" (UID: \"da68f9e7-997a-4697-96c7-c6983f10a42a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:23.829372 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:23.829283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:26.154665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.154637 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq"] Jul 02 20:26:26.156563 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:26.156534 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda68f9e7_997a_4697_96c7_c6983f10a42a.slice/crio-742fc1534dd65cec037c01c624618d906071a3c256c47587129b3b9c3a6e1d32 WatchSource:0}: Error finding container 742fc1534dd65cec037c01c624618d906071a3c256c47587129b3b9c3a6e1d32: Status 404 returned error can't find the container with id 742fc1534dd65cec037c01c624618d906071a3c256c47587129b3b9c3a6e1d32 Jul 02 20:26:26.289113 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.289031 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6"] Jul 02 20:26:26.292480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.292458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:26.295509 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.295489 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kdcmn\"" Jul 02 20:26:26.315142 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.315111 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6"] Jul 02 20:26:26.401554 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.401516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7db\" (UniqueName: \"kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db\") pod \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" (UID: \"15fdf542-6281-4244-ae8b-a1b92779963a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:26.502895 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.502862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7db\" (UniqueName: \"kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db\") pod \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" (UID: \"15fdf542-6281-4244-ae8b-a1b92779963a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:26.523341 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.523313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7db\" (UniqueName: \"kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db\") pod \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" (UID: \"15fdf542-6281-4244-ae8b-a1b92779963a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:26.601938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.601826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:26.748467 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.748432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6"] Jul 02 20:26:26.751365 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:26.751337 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fdf542_6281_4244_ae8b_a1b92779963a.slice/crio-60cfd117e583aa5bc3e96de5d9a83f9cf2186a7716299afcc8bc11ad52e67d4b WatchSource:0}: Error finding container 60cfd117e583aa5bc3e96de5d9a83f9cf2186a7716299afcc8bc11ad52e67d4b: Status 404 returned error can't find the container with id 60cfd117e583aa5bc3e96de5d9a83f9cf2186a7716299afcc8bc11ad52e67d4b Jul 02 20:26:26.890515 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.890480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" event={"ID":"da68f9e7-997a-4697-96c7-c6983f10a42a","Type":"ContainerStarted","Data":"742fc1534dd65cec037c01c624618d906071a3c256c47587129b3b9c3a6e1d32"} Jul 02 20:26:26.892415 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.892383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" event={"ID":"0ff30e57-f092-4ee5-ac9e-9113236b724a","Type":"ContainerStarted","Data":"a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304"} Jul 02 20:26:26.892669 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.892617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:26.893866 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.893780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" event={"ID":"15fdf542-6281-4244-ae8b-a1b92779963a","Type":"ContainerStarted","Data":"60cfd117e583aa5bc3e96de5d9a83f9cf2186a7716299afcc8bc11ad52e67d4b"} Jul 02 20:26:26.895549 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.895516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" event={"ID":"5bde44fc-d3a3-4236-bcc6-eea5acf608c6","Type":"ContainerStarted","Data":"607352efd8856c01a2aaa024e0139565534e1e8b603fd5cd8fb5c770482c76e3"} Jul 02 20:26:26.895705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.895687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:26.932225 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.932167 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" podStartSLOduration=1.5190528730000001 podStartE2EDuration="7.932148259s" totalImagesPullingTime="6.413095386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:26.931455855 +0000 UTC m=+695.136842501" watchObservedRunningTime="2026-07-02 20:26:26.932148259 +0000 UTC m=+695.137534890" Jul 02 20:26:26.969081 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:26.969026 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" podStartSLOduration=1.93218529 podStartE2EDuration="5.969005639s" totalImagesPullingTime="4.036820349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:26.966898418 +0000 UTC m=+695.172285067" watchObservedRunningTime="2026-07-02 20:26:26.969005639 +0000 UTC m=+695.174392268" Jul 02 20:26:28.908066 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.908024 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" event={"ID":"da68f9e7-997a-4697-96c7-c6983f10a42a","Type":"ContainerStarted","Data":"aca9b11a41aadfd28aa410399ba323d309cedf192f41d164467fa58aa82b7c34"} Jul 02 20:26:28.908579 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.908168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:28.910042 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.910019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" event={"ID":"15fdf542-6281-4244-ae8b-a1b92779963a","Type":"ContainerStarted","Data":"b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f"} Jul 02 20:26:28.910155 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.910140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:28.949263 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.949193 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" podStartSLOduration=3.8239487309999998 podStartE2EDuration="5.949170705s" totalImagesPullingTime="2.125221974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:28.946890356 +0000 UTC m=+697.152277002" watchObservedRunningTime="2026-07-02 20:26:28.949170705 +0000 UTC m=+697.154557334" Jul 02 20:26:28.990147 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:28.990091 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" podStartSLOduration=0.960720294 podStartE2EDuration="2.990074786s" totalImagesPullingTime="2.029354492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:28.989803514 +0000 UTC m=+697.195190144" watchObservedRunningTime="2026-07-02 20:26:28.990074786 +0000 UTC m=+697.195461447" Jul 02 20:26:37.903208 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:37.903174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pzxrv" Jul 02 20:26:37.903669 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:37.903242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:39.918138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:39.918104 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qskkq" Jul 02 20:26:39.918518 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:39.918186 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.518195 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.518155 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w"] Jul 02 20:26:40.518420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.518392 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" containerName="manager" containerID="cri-o://a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304" gracePeriod=2 Jul 02 20:26:40.532459 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.532423 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w"] Jul 02 20:26:40.575332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.575295 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6"] Jul 02 20:26:40.575586 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.575514 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" containerName="manager" containerID="cri-o://b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f" gracePeriod=2 Jul 02 20:26:40.578434 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.578393 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.611924 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.611894 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:40.612435 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.612418 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" containerName="manager" Jul 02 20:26:40.612498 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.612441 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" containerName="manager" Jul 02 20:26:40.612537 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.612516 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" containerName="manager" Jul 02 20:26:40.618920 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.618896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.626525 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.626488 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6"] Jul 02 20:26:40.629609 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.629563 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"15fdf542-6281-4244-ae8b-a1b92779963a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-xcqf6\": pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.639821 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.639790 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6"] Jul 02 20:26:40.640158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.640145 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" containerName="manager" Jul 02 20:26:40.640200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.640160 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" containerName="manager" Jul 02 20:26:40.640253 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.640243 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" containerName="manager" Jul 02 20:26:40.643066 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.643043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:40.655965 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.655936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:40.717757 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.715663 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.721242 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.720532 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6"] Jul 02 20:26:40.723294 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.723200 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.727708 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.727677 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.735160 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.735131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.735318 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.735184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh96g\" (UniqueName: \"kubernetes.io/projected/6ce73322-ad0c-49d7-b489-47ce3b7798b6-kube-api-access-nh96g\") pod \"limitador-operator-controller-manager-859d7b78bd-65sz6\" (UID: \"6ce73322-ad0c-49d7-b489-47ce3b7798b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:40.735318 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.735223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9b6\" (UniqueName: \"kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.737922 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.737885 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.823100 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.823069 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.825725 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.825699 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.826022 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.826007 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.827890 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.827866 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.830251 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.830229 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.835370 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.835344 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.835648 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.835631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.835701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.835664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh96g\" (UniqueName: \"kubernetes.io/projected/6ce73322-ad0c-49d7-b489-47ce3b7798b6-kube-api-access-nh96g\") pod \"limitador-operator-controller-manager-859d7b78bd-65sz6\" (UID: \"6ce73322-ad0c-49d7-b489-47ce3b7798b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:40.835701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.835690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9b6\" (UniqueName: \"kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.836053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.836027 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.851148 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.851126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9b6\" (UniqueName: \"kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2s554\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.855269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.855251 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh96g\" (UniqueName: \"kubernetes.io/projected/6ce73322-ad0c-49d7-b489-47ce3b7798b6-kube-api-access-nh96g\") pod \"limitador-operator-controller-manager-859d7b78bd-65sz6\" (UID: \"6ce73322-ad0c-49d7-b489-47ce3b7798b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:40.936455 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.936414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm\") pod \"0ff30e57-f092-4ee5-ac9e-9113236b724a\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " Jul 02 20:26:40.936814 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.936470 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk7db\" (UniqueName: \"kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db\") pod \"15fdf542-6281-4244-ae8b-a1b92779963a\" (UID: \"15fdf542-6281-4244-ae8b-a1b92779963a\") " Jul 02 20:26:40.936814 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.936558 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume\") pod \"0ff30e57-f092-4ee5-ac9e-9113236b724a\" (UID: \"0ff30e57-f092-4ee5-ac9e-9113236b724a\") " Jul 02 20:26:40.937046 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.937025 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ff30e57-f092-4ee5-ac9e-9113236b724a" (UID: "0ff30e57-f092-4ee5-ac9e-9113236b724a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:26:40.938677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.938657 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db" (OuterVolumeSpecName: "kube-api-access-lk7db") pod "15fdf542-6281-4244-ae8b-a1b92779963a" (UID: "15fdf542-6281-4244-ae8b-a1b92779963a"). InnerVolumeSpecName "kube-api-access-lk7db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:26:40.938731 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.938681 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm" (OuterVolumeSpecName: "kube-api-access-9nwlm") pod "0ff30e57-f092-4ee5-ac9e-9113236b724a" (UID: "0ff30e57-f092-4ee5-ac9e-9113236b724a"). InnerVolumeSpecName "kube-api-access-9nwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:26:40.962190 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.962160 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:40.963493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.963468 2579 generic.go:356] "Generic (PLEG): container finished" podID="15fdf542-6281-4244-ae8b-a1b92779963a" containerID="b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f" exitCode=0 Jul 02 20:26:40.963596 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.963516 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.963596 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.963574 2579 scope.go:117] "RemoveContainer" containerID="b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f" Jul 02 20:26:40.965229 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.965111 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ff30e57-f092-4ee5-ac9e-9113236b724a" containerID="a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304" exitCode=0 Jul 02 20:26:40.965229 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.965160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.972653 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.972607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:40.973033 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.972998 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.975666 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.975638 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.977465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.977272 2579 scope.go:117] "RemoveContainer" containerID="b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f" Jul 02 20:26:40.977675 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.977643 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f\": container with ID starting with b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f not found: ID does not exist" containerID="b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f" Jul 02 20:26:40.977748 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.977680 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f"} err="failed to get container status \"b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f\": rpc error: code = NotFound desc = could not find container \"b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f\": container with ID starting with b0a12b7755abb1e509b29efbe4a95930cf1c8f3af6ce646d443429a424d5f87f not found: ID does not exist" Jul 02 20:26:40.977748 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.977703 2579 scope.go:117] "RemoveContainer" containerID="a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304" Jul 02 20:26:40.978067 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.978047 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.980864 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.980819 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.984341 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.984309 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:40.987342 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.987317 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:40.987737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.987713 2579 scope.go:117] "RemoveContainer" containerID="a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304" Jul 02 20:26:40.988059 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:40.988038 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304\": container with ID starting with a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304 not found: ID does not exist" containerID="a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304" Jul 02 20:26:40.988146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:40.988065 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304"} err="failed to get container status \"a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304\": rpc error: code = NotFound desc = could not find container \"a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304\": container with ID starting with a402dab1e41c7a8ba57c4fb0352dd055cd2dac6328355406070f2b6d51334304 not found: ID does not exist" Jul 02 20:26:41.037826 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.037750 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ff30e57-f092-4ee5-ac9e-9113236b724a-extensions-socket-volume\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:26:41.037826 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.037785 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/0ff30e57-f092-4ee5-ac9e-9113236b724a-kube-api-access-9nwlm\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:26:41.037826 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.037800 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7db\" (UniqueName: \"kubernetes.io/projected/15fdf542-6281-4244-ae8b-a1b92779963a-kube-api-access-lk7db\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:26:41.162316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.162289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:41.165015 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:41.164989 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9724fdde_4e2d_45fe_bb49_a7270994344f.slice/crio-59dea8b13f082d52239a77567ea11fb3687b6356329b7f37670c368e2dc4f891 WatchSource:0}: Error finding container 59dea8b13f082d52239a77567ea11fb3687b6356329b7f37670c368e2dc4f891: Status 404 returned error can't find the container with id 59dea8b13f082d52239a77567ea11fb3687b6356329b7f37670c368e2dc4f891 Jul 02 20:26:41.175671 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.175645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6"] Jul 02 20:26:41.181777 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:41.181757 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce73322_ad0c_49d7_b489_47ce3b7798b6.slice/crio-f486777ccbc61935a9bfc648bbbff2a2c6426cdcef1a9503d1ab2d6ae5924d7b WatchSource:0}: Error finding container f486777ccbc61935a9bfc648bbbff2a2c6426cdcef1a9503d1ab2d6ae5924d7b: Status 404 returned error can't find the container with id f486777ccbc61935a9bfc648bbbff2a2c6426cdcef1a9503d1ab2d6ae5924d7b Jul 02 20:26:41.971929 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.971889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" event={"ID":"6ce73322-ad0c-49d7-b489-47ce3b7798b6","Type":"ContainerStarted","Data":"f81dd75d0fbec3c7393f885750454b1b671b22824a13ec2dd80714f67ca8ffcb"} Jul 02 20:26:41.972373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.971959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" event={"ID":"6ce73322-ad0c-49d7-b489-47ce3b7798b6","Type":"ContainerStarted","Data":"f486777ccbc61935a9bfc648bbbff2a2c6426cdcef1a9503d1ab2d6ae5924d7b"} Jul 02 20:26:41.972373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.972007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:41.973802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.973778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" event={"ID":"9724fdde-4e2d-45fe-bb49-a7270994344f","Type":"ContainerStarted","Data":"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982"} Jul 02 20:26:41.973963 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.973816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" event={"ID":"9724fdde-4e2d-45fe-bb49-a7270994344f","Type":"ContainerStarted","Data":"59dea8b13f082d52239a77567ea11fb3687b6356329b7f37670c368e2dc4f891"} Jul 02 20:26:41.973963 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:41.973892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:41.976568 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:41.976543 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:41.979441 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:41.979421 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:42.018640 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:42.018593 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" podStartSLOduration=2.018579463 podStartE2EDuration="2.018579463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:42.01667622 +0000 UTC m=+710.222062859" watchObservedRunningTime="2026-07-02 20:26:42.018579463 +0000 UTC m=+710.223966093" Jul 02 20:26:42.019334 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:42.019313 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:42.023092 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:42.023074 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:42.085063 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:42.085009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" podStartSLOduration=2.084993038 podStartE2EDuration="2.084993038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:42.084397604 +0000 UTC m=+710.289784254" watchObservedRunningTime="2026-07-02 20:26:42.084993038 +0000 UTC m=+710.290379667" Jul 02 20:26:42.339722 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:42.339637 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7xr2w\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7xr2w" Jul 02 20:26:42.341038 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:42.341017 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff30e57-f092-4ee5-ac9e-9113236b724a" path="/var/lib/kubelet/pods/0ff30e57-f092-4ee5-ac9e-9113236b724a/volumes" Jul 02 20:26:42.341317 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:42.341305 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" path="/var/lib/kubelet/pods/15fdf542-6281-4244-ae8b-a1b92779963a/volumes" Jul 02 20:26:42.345549 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:42.345528 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xcqf6\" is forbidden: User \"system:node:ip-10-0-139-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-106.ec2.internal' and this object" podUID="15fdf542-6281-4244-ae8b-a1b92779963a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xcqf6" Jul 02 20:26:43.768965 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.768934 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:26:43.773277 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.773261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.801634 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.801610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:26:43.864246 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.864210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.864452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.864264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5w4\" (UniqueName: \"kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.964789 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.964745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.964789 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.964791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5w4\" (UniqueName: \"kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.965140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.965120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:43.982585 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:43.982563 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5w4\" (UniqueName: \"kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4\") pod \"kuadrant-operator-controller-manager-86878c769f-tc78w\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:44.083198 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:44.083118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:44.236613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:44.236581 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:26:44.240115 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:26:44.240086 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40614eb9_e1d2_4697_968d_e761abc73c0e.slice/crio-e8a65257ad387b222e6b62aa440982cf411123e53567cfd170e56a045dfec427 WatchSource:0}: Error finding container e8a65257ad387b222e6b62aa440982cf411123e53567cfd170e56a045dfec427: Status 404 returned error can't find the container with id e8a65257ad387b222e6b62aa440982cf411123e53567cfd170e56a045dfec427 Jul 02 20:26:44.991562 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:44.991530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" event={"ID":"40614eb9-e1d2-4697-968d-e761abc73c0e","Type":"ContainerStarted","Data":"46c9380815cc6b5e0d5b020a8bc30187d9b7617c0abcd895f6ff08ce4b7c12ce"} Jul 02 20:26:44.991957 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:44.991581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" event={"ID":"40614eb9-e1d2-4697-968d-e761abc73c0e","Type":"ContainerStarted","Data":"e8a65257ad387b222e6b62aa440982cf411123e53567cfd170e56a045dfec427"} Jul 02 20:26:44.991957 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:44.991656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:45.015197 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:45.015151 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" podStartSLOduration=2.015138945 podStartE2EDuration="2.015138945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:45.013632431 +0000 UTC m=+713.219019061" watchObservedRunningTime="2026-07-02 20:26:45.015138945 +0000 UTC m=+713.220525574" Jul 02 20:26:52.982529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:52.982498 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:52.982990 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:52.982651 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65sz6" Jul 02 20:26:56.000282 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.000255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:26:56.063385 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.063353 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:56.063630 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.063606 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" podUID="9724fdde-4e2d-45fe-bb49-a7270994344f" containerName="manager" containerID="cri-o://dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982" gracePeriod=10 Jul 02 20:26:56.326011 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.325984 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:56.476331 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.476294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b9b6\" (UniqueName: \"kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6\") pod \"9724fdde-4e2d-45fe-bb49-a7270994344f\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " Jul 02 20:26:56.476511 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.476356 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume\") pod \"9724fdde-4e2d-45fe-bb49-a7270994344f\" (UID: \"9724fdde-4e2d-45fe-bb49-a7270994344f\") " Jul 02 20:26:56.476780 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.476750 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9724fdde-4e2d-45fe-bb49-a7270994344f" (UID: "9724fdde-4e2d-45fe-bb49-a7270994344f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:26:56.478540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.478519 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6" (OuterVolumeSpecName: "kube-api-access-8b9b6") pod "9724fdde-4e2d-45fe-bb49-a7270994344f" (UID: "9724fdde-4e2d-45fe-bb49-a7270994344f"). InnerVolumeSpecName "kube-api-access-8b9b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:26:56.577885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.577781 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9b6\" (UniqueName: \"kubernetes.io/projected/9724fdde-4e2d-45fe-bb49-a7270994344f-kube-api-access-8b9b6\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:26:56.577885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:56.577811 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9724fdde-4e2d-45fe-bb49-a7270994344f-extensions-socket-volume\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:26:57.052825 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.052794 2579 generic.go:356] "Generic (PLEG): container finished" podID="9724fdde-4e2d-45fe-bb49-a7270994344f" containerID="dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982" exitCode=0 Jul 02 20:26:57.053241 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.052881 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" Jul 02 20:26:57.053241 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.052879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" event={"ID":"9724fdde-4e2d-45fe-bb49-a7270994344f","Type":"ContainerDied","Data":"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982"} Jul 02 20:26:57.053241 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.052935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554" event={"ID":"9724fdde-4e2d-45fe-bb49-a7270994344f","Type":"ContainerDied","Data":"59dea8b13f082d52239a77567ea11fb3687b6356329b7f37670c368e2dc4f891"} Jul 02 20:26:57.053241 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.052956 2579 scope.go:117] "RemoveContainer" containerID="dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982" Jul 02 20:26:57.061934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.061917 2579 scope.go:117] "RemoveContainer" containerID="dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982" Jul 02 20:26:57.062203 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:26:57.062183 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982\": container with ID starting with dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982 not found: ID does not exist" containerID="dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982" Jul 02 20:26:57.062305 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.062208 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982"} err="failed to get container status \"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982\": rpc error: code = NotFound desc = could not find container \"dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982\": container with ID starting with dbf360fb2c9404726293ff2c4c5f3840f89fa49780ee0882fc207ce8f63a6982 not found: ID does not exist" Jul 02 20:26:57.080236 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.080213 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:57.082236 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:57.082217 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2s554"] Jul 02 20:26:58.339768 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:26:58.339726 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9724fdde-4e2d-45fe-bb49-a7270994344f" path="/var/lib/kubelet/pods/9724fdde-4e2d-45fe-bb49-a7270994344f/volumes" Jul 02 20:27:12.657159 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.657123 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw"] Jul 02 20:27:12.657526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.657447 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9724fdde-4e2d-45fe-bb49-a7270994344f" containerName="manager" Jul 02 20:27:12.657526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.657458 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9724fdde-4e2d-45fe-bb49-a7270994344f" containerName="manager" Jul 02 20:27:12.657526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.657526 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9724fdde-4e2d-45fe-bb49-a7270994344f" containerName="manager" Jul 02 20:27:12.661942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.661919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.666128 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.666090 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 20:27:12.666228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.666155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r94dg\"" Jul 02 20:27:12.693275 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.693226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw"] Jul 02 20:27:12.715791 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b805a60f-d88c-4b41-80a0-05e23ccb1451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.715791 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq28c\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-kube-api-access-wq28c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716007 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.715984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.716029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.716158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.716067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817064 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b805a60f-d88c-4b41-80a0-05e23ccb1451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq28c\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-kube-api-access-wq28c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817481 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817638 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817860 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817953 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.817953 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.817885 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b805a60f-d88c-4b41-80a0-05e23ccb1451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.819714 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.819697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.819774 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.819725 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.833395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.833374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq28c\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-kube-api-access-wq28c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.836806 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.836788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b805a60f-d88c-4b41-80a0-05e23ccb1451-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-k5kdw\" (UID: \"b805a60f-d88c-4b41-80a0-05e23ccb1451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:12.972072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:12.971978 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:13.125613 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:13.125583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw"] Jul 02 20:27:13.133402 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:27:13.133372 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb805a60f_d88c_4b41_80a0_05e23ccb1451.slice/crio-c52ea64f1fc612ac40b1c21845deca87f811053a6174a263561a08c0eadf5f7e WatchSource:0}: Error finding container c52ea64f1fc612ac40b1c21845deca87f811053a6174a263561a08c0eadf5f7e: Status 404 returned error can't find the container with id c52ea64f1fc612ac40b1c21845deca87f811053a6174a263561a08c0eadf5f7e Jul 02 20:27:14.134505 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:14.134462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" event={"ID":"b805a60f-d88c-4b41-80a0-05e23ccb1451","Type":"ContainerStarted","Data":"c52ea64f1fc612ac40b1c21845deca87f811053a6174a263561a08c0eadf5f7e"} Jul 02 20:27:15.657898 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:15.657852 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:27:15.658191 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:15.657924 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:27:15.658191 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:15.657950 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:27:16.145519 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:16.145485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" event={"ID":"b805a60f-d88c-4b41-80a0-05e23ccb1451","Type":"ContainerStarted","Data":"5ffd37105ad03a4a00f24d42797a4e5c93d3d08ef797e0abd324f8f0e92af9b1"} Jul 02 20:27:16.176075 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:16.176018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" podStartSLOduration=1.653691952 podStartE2EDuration="4.175996s" totalImagesPullingTime="2.522304048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:16.17565532 +0000 UTC m=+744.381041951" watchObservedRunningTime="2026-07-02 20:27:16.175996 +0000 UTC m=+744.381382630" Jul 02 20:27:16.973158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:16.973118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:16.978075 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:16.978044 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:17.150228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.150199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:17.151053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.151033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:27:17.711407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.711369 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:27:17.719463 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.719434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.722588 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.722570 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hzkvd\"" Jul 02 20:27:17.725888 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.725756 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 20:27:17.726230 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.726207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:27:17.760004 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.759966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qfx\" (UniqueName: \"kubernetes.io/projected/a0d16f89-a00d-4eb8-9ac7-678858640de7-kube-api-access-x6qfx\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.760186 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.760029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d16f89-a00d-4eb8-9ac7-678858640de7-config-file\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.765664 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.765631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:27:17.860734 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.860694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d16f89-a00d-4eb8-9ac7-678858640de7-config-file\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.860944 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.860797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qfx\" (UniqueName: \"kubernetes.io/projected/a0d16f89-a00d-4eb8-9ac7-678858640de7-kube-api-access-x6qfx\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.861381 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.861362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0d16f89-a00d-4eb8-9ac7-678858640de7-config-file\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:17.870999 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:17.870974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qfx\" (UniqueName: \"kubernetes.io/projected/a0d16f89-a00d-4eb8-9ac7-678858640de7-kube-api-access-x6qfx\") pod \"limitador-limitador-54d6d988d5-fnx8p\" (UID: \"a0d16f89-a00d-4eb8-9ac7-678858640de7\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:18.031358 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.031266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:18.191702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.191539 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:27:18.194407 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:27:18.194380 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d16f89_a00d_4eb8_9ac7_678858640de7.slice/crio-76fdbcecc8ce8b0e92a41efadba1100fe3042744eb0ddba29db34eb2c08229ea WatchSource:0}: Error finding container 76fdbcecc8ce8b0e92a41efadba1100fe3042744eb0ddba29db34eb2c08229ea: Status 404 returned error can't find the container with id 76fdbcecc8ce8b0e92a41efadba1100fe3042744eb0ddba29db34eb2c08229ea Jul 02 20:27:18.656110 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.656069 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:18.658966 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.658941 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:18.662013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.661990 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ttq9h\"" Jul 02 20:27:18.672004 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.671979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:18.771978 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.771776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584lr\" (UniqueName: \"kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr\") pod \"authorino-75d556bd77-zr5wh\" (UID: \"099f2fc4-2807-43f0-8d43-408b90431f1a\") " pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:18.873278 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.872790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584lr\" (UniqueName: \"kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr\") pod \"authorino-75d556bd77-zr5wh\" (UID: \"099f2fc4-2807-43f0-8d43-408b90431f1a\") " pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:18.887975 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.887794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-584lr\" (UniqueName: \"kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr\") pod \"authorino-75d556bd77-zr5wh\" (UID: \"099f2fc4-2807-43f0-8d43-408b90431f1a\") " pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:18.973900 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:18.973806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:19.140291 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:19.139906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:19.143470 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:27:19.143443 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod099f2fc4_2807_43f0_8d43_408b90431f1a.slice/crio-7beb6952ab2b5c14680bf45843e6900572d5dbd6cd767d17d6ee33b49d8867b5 WatchSource:0}: Error finding container 7beb6952ab2b5c14680bf45843e6900572d5dbd6cd767d17d6ee33b49d8867b5: Status 404 returned error can't find the container with id 7beb6952ab2b5c14680bf45843e6900572d5dbd6cd767d17d6ee33b49d8867b5 Jul 02 20:27:19.161695 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:19.161652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zr5wh" event={"ID":"099f2fc4-2807-43f0-8d43-408b90431f1a","Type":"ContainerStarted","Data":"7beb6952ab2b5c14680bf45843e6900572d5dbd6cd767d17d6ee33b49d8867b5"} Jul 02 20:27:19.163117 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:19.163091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" event={"ID":"a0d16f89-a00d-4eb8-9ac7-678858640de7","Type":"ContainerStarted","Data":"76fdbcecc8ce8b0e92a41efadba1100fe3042744eb0ddba29db34eb2c08229ea"} Jul 02 20:27:21.176290 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:21.176049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" event={"ID":"a0d16f89-a00d-4eb8-9ac7-678858640de7","Type":"ContainerStarted","Data":"4cf7ecfee32dd7ff91cbb8d62fad06ef145aa81def395e399b325c84953498fe"} Jul 02 20:27:21.176290 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:21.176173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:24.192279 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:24.192236 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zr5wh" event={"ID":"099f2fc4-2807-43f0-8d43-408b90431f1a","Type":"ContainerStarted","Data":"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d"} Jul 02 20:27:24.213379 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:24.213335 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zr5wh" podStartSLOduration=2.179555468 podStartE2EDuration="6.213320769s" totalImagesPullingTime="4.033765301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:24.209378881 +0000 UTC m=+752.414765527" watchObservedRunningTime="2026-07-02 20:27:24.213320769 +0000 UTC m=+752.418707397" Jul 02 20:27:24.213754 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:24.213734 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" podStartSLOduration=4.666142518 podStartE2EDuration="7.213726232s" totalImagesPullingTime="2.547583714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:21.196894763 +0000 UTC m=+749.402281417" watchObservedRunningTime="2026-07-02 20:27:24.213726232 +0000 UTC m=+752.419112858" Jul 02 20:27:32.183148 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:32.183118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fnx8p" Jul 02 20:27:47.744279 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.744243 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:27:47.747779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.747755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:47.750775 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.750758 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 02 20:27:47.758756 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.758734 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:27:47.927242 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.927199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:47.927417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:47.927281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwgv\" (UniqueName: \"kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.027819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.027722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.028009 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.027823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwgv\" (UniqueName: \"kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.030385 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.030366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.039862 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.039825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwgv\" (UniqueName: \"kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv\") pod \"authorino-5f7979d584-b8rpk\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.056825 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.056789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:27:48.217360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.217331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:27:48.219559 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:27:48.219534 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc7ab3a1_3965_4aee_b5a9_7e6e0f5f3aba.slice/crio-9efc87a77c09220b39ca627e2774c3901f60016abc1d56a8c60caed919efa479 WatchSource:0}: Error finding container 9efc87a77c09220b39ca627e2774c3901f60016abc1d56a8c60caed919efa479: Status 404 returned error can't find the container with id 9efc87a77c09220b39ca627e2774c3901f60016abc1d56a8c60caed919efa479 Jul 02 20:27:48.301770 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:48.301689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7979d584-b8rpk" event={"ID":"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba","Type":"ContainerStarted","Data":"9efc87a77c09220b39ca627e2774c3901f60016abc1d56a8c60caed919efa479"} Jul 02 20:27:49.308256 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.308218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7979d584-b8rpk" event={"ID":"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba","Type":"ContainerStarted","Data":"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a"} Jul 02 20:27:49.373018 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.372955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f7979d584-b8rpk" podStartSLOduration=1.8966524630000001 podStartE2EDuration="2.372925285s" totalImagesPullingTime="476.272822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:49.335418119 +0000 UTC m=+777.540804747" watchObservedRunningTime="2026-07-02 20:27:49.372925285 +0000 UTC m=+777.578311914" Jul 02 20:27:49.375277 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.375247 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:49.375659 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.375631 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zr5wh" podUID="099f2fc4-2807-43f0-8d43-408b90431f1a" containerName="authorino" containerID="cri-o://9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d" gracePeriod=30 Jul 02 20:27:49.657461 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.657430 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:49.843867 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.843813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584lr\" (UniqueName: \"kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr\") pod \"099f2fc4-2807-43f0-8d43-408b90431f1a\" (UID: \"099f2fc4-2807-43f0-8d43-408b90431f1a\") " Jul 02 20:27:49.845948 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.845917 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr" (OuterVolumeSpecName: "kube-api-access-584lr") pod "099f2fc4-2807-43f0-8d43-408b90431f1a" (UID: "099f2fc4-2807-43f0-8d43-408b90431f1a"). InnerVolumeSpecName "kube-api-access-584lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:27:49.944875 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:49.944769 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584lr\" (UniqueName: \"kubernetes.io/projected/099f2fc4-2807-43f0-8d43-408b90431f1a-kube-api-access-584lr\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:27:50.314002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.313913 2579 generic.go:356] "Generic (PLEG): container finished" podID="099f2fc4-2807-43f0-8d43-408b90431f1a" containerID="9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d" exitCode=0 Jul 02 20:27:50.314002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.313967 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zr5wh" Jul 02 20:27:50.314417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.314004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zr5wh" event={"ID":"099f2fc4-2807-43f0-8d43-408b90431f1a","Type":"ContainerDied","Data":"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d"} Jul 02 20:27:50.314417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.314048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zr5wh" event={"ID":"099f2fc4-2807-43f0-8d43-408b90431f1a","Type":"ContainerDied","Data":"7beb6952ab2b5c14680bf45843e6900572d5dbd6cd767d17d6ee33b49d8867b5"} Jul 02 20:27:50.314417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.314070 2579 scope.go:117] "RemoveContainer" containerID="9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d" Jul 02 20:27:50.323985 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.323966 2579 scope.go:117] "RemoveContainer" containerID="9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d" Jul 02 20:27:50.324288 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:27:50.324270 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d\": container with ID starting with 9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d not found: ID does not exist" containerID="9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d" Jul 02 20:27:50.324336 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.324300 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d"} err="failed to get container status \"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d\": rpc error: code = NotFound desc = could not find container \"9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d\": container with ID starting with 9f1d953ffda0bbed353818d98ce75bef6f1e1738df00f94c9ce23fce2a3c7b6d not found: ID does not exist" Jul 02 20:27:50.340938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.340914 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:50.344249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:50.344228 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zr5wh"] Jul 02 20:27:52.339987 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:27:52.339958 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099f2fc4-2807-43f0-8d43-408b90431f1a" path="/var/lib/kubelet/pods/099f2fc4-2807-43f0-8d43-408b90431f1a/volumes" Jul 02 20:28:32.379862 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:32.379810 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:28:40.545477 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.545445 2579 generic.go:356] "Generic (PLEG): container finished" podID="b805a60f-d88c-4b41-80a0-05e23ccb1451" containerID="5ffd37105ad03a4a00f24d42797a4e5c93d3d08ef797e0abd324f8f0e92af9b1" exitCode=0 Jul 02 20:28:40.545914 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.545530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" event={"ID":"b805a60f-d88c-4b41-80a0-05e23ccb1451","Type":"ContainerDied","Data":"5ffd37105ad03a4a00f24d42797a4e5c93d3d08ef797e0abd324f8f0e92af9b1"} Jul 02 20:28:40.545959 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.545934 2579 scope.go:117] "RemoveContainer" containerID="5ffd37105ad03a4a00f24d42797a4e5c93d3d08ef797e0abd324f8f0e92af9b1" Jul 02 20:28:40.547040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.547004 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:28:40.547161 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.547085 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:28:40.547161 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:40.547130 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:28:41.554863 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:41.554798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" event={"ID":"b805a60f-d88c-4b41-80a0-05e23ccb1451","Type":"ContainerStarted","Data":"10df8153df377adaec7c6539fb4506093980defe0401b6f6597e7e01de4b0aa3"} Jul 02 20:28:41.973161 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:41.973117 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:28:41.973341 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:41.973321 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-k5kdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 02 20:28:41.973383 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:41.973365 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" podUID="b805a60f-d88c-4b41-80a0-05e23ccb1451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 02 20:28:42.972568 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:42.972525 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-k5kdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 02 20:28:42.973050 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:42.972590 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" podUID="b805a60f-d88c-4b41-80a0-05e23ccb1451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 02 20:28:43.973183 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:43.973139 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-k5kdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 02 20:28:43.973647 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:43.973197 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" podUID="b805a60f-d88c-4b41-80a0-05e23ccb1451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 02 20:28:44.972859 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:44.972808 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-k5kdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 02 20:28:44.973055 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:44.972886 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" podUID="b805a60f-d88c-4b41-80a0-05e23ccb1451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 02 20:28:45.976798 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:45.976771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:28:45.977187 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:45.977125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:28:45.977735 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:45.977718 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-k5kdw" Jul 02 20:28:57.935271 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:28:57.935237 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:09.901748 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:09.901715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:20.394289 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:20.394253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:22.294907 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.294872 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp"] Jul 02 20:29:22.295475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.295459 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099f2fc4-2807-43f0-8d43-408b90431f1a" containerName="authorino" Jul 02 20:29:22.295528 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.295479 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="099f2fc4-2807-43f0-8d43-408b90431f1a" containerName="authorino" Jul 02 20:29:22.295594 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.295583 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="099f2fc4-2807-43f0-8d43-408b90431f1a" containerName="authorino" Jul 02 20:29:22.298010 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.297993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.300614 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.300593 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 20:29:22.301694 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.301677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xgxfv\"" Jul 02 20:29:22.301813 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.301717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 20:29:22.301813 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.301735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 02 20:29:22.307797 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.307769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp"] Jul 02 20:29:22.355810 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.355750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.356015 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.355818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.356015 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.355887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca713c6-1c7d-4cbe-9d97-94696973339e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.356015 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.355944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdh52\" (UniqueName: \"kubernetes.io/projected/1ca713c6-1c7d-4cbe-9d97-94696973339e-kube-api-access-rdh52\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.356015 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.356011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.356250 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.356092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456724 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca713c6-1c7d-4cbe-9d97-94696973339e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.456922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.456866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdh52\" (UniqueName: \"kubernetes.io/projected/1ca713c6-1c7d-4cbe-9d97-94696973339e-kube-api-access-rdh52\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.457202 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.457187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.457271 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.457190 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.457271 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.457247 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.459161 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.459137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca713c6-1c7d-4cbe-9d97-94696973339e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.459468 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.459448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca713c6-1c7d-4cbe-9d97-94696973339e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.466191 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.466164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdh52\" (UniqueName: \"kubernetes.io/projected/1ca713c6-1c7d-4cbe-9d97-94696973339e-kube-api-access-rdh52\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp\" (UID: \"1ca713c6-1c7d-4cbe-9d97-94696973339e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.610231 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.610200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:22.748626 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.748598 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp"] Jul 02 20:29:22.750252 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:29:22.750217 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca713c6_1c7d_4cbe_9d97_94696973339e.slice/crio-bdd226f6f7bc1015c613b4c8d75b79999122466ff84e985dc898ced9a5822fd0 WatchSource:0}: Error finding container bdd226f6f7bc1015c613b4c8d75b79999122466ff84e985dc898ced9a5822fd0: Status 404 returned error can't find the container with id bdd226f6f7bc1015c613b4c8d75b79999122466ff84e985dc898ced9a5822fd0 Jul 02 20:29:22.760878 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:22.760850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" event={"ID":"1ca713c6-1c7d-4cbe-9d97-94696973339e","Type":"ContainerStarted","Data":"bdd226f6f7bc1015c613b4c8d75b79999122466ff84e985dc898ced9a5822fd0"} Jul 02 20:29:24.772400 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:24.772356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" event={"ID":"1ca713c6-1c7d-4cbe-9d97-94696973339e","Type":"ContainerStarted","Data":"4df2b8c41520a16fbeb608d629c45f7580c2d6aef9d8d8644344de79a1d9ad70"} Jul 02 20:29:24.792493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:24.792434 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" podStartSLOduration=1.767866294 podStartE2EDuration="2.792380015s" totalImagesPullingTime="1.024513721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:29:24.791701963 +0000 UTC m=+872.997088608" watchObservedRunningTime="2026-07-02 20:29:24.792380015 +0000 UTC m=+872.997766646" Jul 02 20:29:27.699938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:27.699900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:32.610892 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:32.610830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:32.611280 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:32.610953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:32.623339 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:32.623312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:32.826207 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:32.826176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp" Jul 02 20:29:33.501904 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:33.501868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:41.209632 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:41.209591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:29:56.117332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:56.117302 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:29:56.117808 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:29:56.117368 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:30:29.272509 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.272470 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc4b4dc57-xpbmv"] Jul 02 20:30:29.275909 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.275891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.283129 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.283055 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc4b4dc57-xpbmv"] Jul 02 20:30:29.360611 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.360573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca470f80-7437-4b4c-af5e-3c0137405e6a-tls-cert\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.360782 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.360630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfhb\" (UniqueName: \"kubernetes.io/projected/ca470f80-7437-4b4c-af5e-3c0137405e6a-kube-api-access-8pfhb\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.461436 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.461396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca470f80-7437-4b4c-af5e-3c0137405e6a-tls-cert\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.461597 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.461449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfhb\" (UniqueName: \"kubernetes.io/projected/ca470f80-7437-4b4c-af5e-3c0137405e6a-kube-api-access-8pfhb\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.464178 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.464144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca470f80-7437-4b4c-af5e-3c0137405e6a-tls-cert\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.470008 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.469981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfhb\" (UniqueName: \"kubernetes.io/projected/ca470f80-7437-4b4c-af5e-3c0137405e6a-kube-api-access-8pfhb\") pod \"authorino-7cc4b4dc57-xpbmv\" (UID: \"ca470f80-7437-4b4c-af5e-3c0137405e6a\") " pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.585273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.585176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" Jul 02 20:30:29.719991 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:29.719961 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc4b4dc57-xpbmv"] Jul 02 20:30:29.722519 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:30:29.722473 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca470f80_7437_4b4c_af5e_3c0137405e6a.slice/crio-c085db7d70fe8285ed181c3c8934ae348fa960ab4e63d4b5e5499267b06cac43 WatchSource:0}: Error finding container c085db7d70fe8285ed181c3c8934ae348fa960ab4e63d4b5e5499267b06cac43: Status 404 returned error can't find the container with id c085db7d70fe8285ed181c3c8934ae348fa960ab4e63d4b5e5499267b06cac43 Jul 02 20:30:30.093006 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:30.092960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" event={"ID":"ca470f80-7437-4b4c-af5e-3c0137405e6a","Type":"ContainerStarted","Data":"c085db7d70fe8285ed181c3c8934ae348fa960ab4e63d4b5e5499267b06cac43"} Jul 02 20:30:31.100637 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.100600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" event={"ID":"ca470f80-7437-4b4c-af5e-3c0137405e6a","Type":"ContainerStarted","Data":"40eaaeca3ffe2058c6f24f23155dc8df76ec425516d19637e079959d36895c40"} Jul 02 20:30:31.118885 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.118800 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cc4b4dc57-xpbmv" podStartSLOduration=1.711590454 podStartE2EDuration="2.118780301s" totalImagesPullingTime="407.189847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:30:31.118251693 +0000 UTC m=+939.323638323" watchObservedRunningTime="2026-07-02 20:30:31.118780301 +0000 UTC m=+939.324166930" Jul 02 20:30:31.153311 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.152018 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:30:31.153311 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.152245 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5f7979d584-b8rpk" podUID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" containerName="authorino" containerID="cri-o://2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a" gracePeriod=30 Jul 02 20:30:31.406913 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.406881 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:30:31.482731 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.482699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert\") pod \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " Jul 02 20:30:31.482936 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.482759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwgv\" (UniqueName: \"kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv\") pod \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\" (UID: \"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba\") " Jul 02 20:30:31.485111 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.485074 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv" (OuterVolumeSpecName: "kube-api-access-sbwgv") pod "fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" (UID: "fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba"). InnerVolumeSpecName "kube-api-access-sbwgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:30:31.493144 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.493112 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" (UID: "fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:30:31.584401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.584362 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-tls-cert\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:30:31.584401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:31.584396 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwgv\" (UniqueName: \"kubernetes.io/projected/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba-kube-api-access-sbwgv\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:30:32.107225 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.107194 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" containerID="2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a" exitCode=0 Jul 02 20:30:32.107705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.107234 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f7979d584-b8rpk" Jul 02 20:30:32.107705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.107263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7979d584-b8rpk" event={"ID":"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba","Type":"ContainerDied","Data":"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a"} Jul 02 20:30:32.107705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.107291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f7979d584-b8rpk" event={"ID":"fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba","Type":"ContainerDied","Data":"9efc87a77c09220b39ca627e2774c3901f60016abc1d56a8c60caed919efa479"} Jul 02 20:30:32.107705 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.107307 2579 scope.go:117] "RemoveContainer" containerID="2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a" Jul 02 20:30:32.117172 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.117155 2579 scope.go:117] "RemoveContainer" containerID="2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a" Jul 02 20:30:32.117485 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:30:32.117463 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a\": container with ID starting with 2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a not found: ID does not exist" containerID="2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a" Jul 02 20:30:32.117579 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.117491 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a"} err="failed to get container status \"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a\": rpc error: code = NotFound desc = could not find container \"2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a\": container with ID starting with 2467bb95912bdfe9121f31d0b336f5bd37e1b8e0f560c4dfb11166740e8fa39a not found: ID does not exist" Jul 02 20:30:32.129060 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.129028 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:30:32.135013 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.134981 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f7979d584-b8rpk"] Jul 02 20:30:32.349802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:30:32.349766 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" path="/var/lib/kubelet/pods/fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba/volumes" Jul 02 20:31:06.806344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:31:06.806305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:31:16.196113 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:31:16.196083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:31:27.207027 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:31:27.206989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:31:36.098363 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:31:36.098328 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:31:47.122576 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:31:47.122542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:32:13.806360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:32:13.806323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:32:25.428225 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:32:25.428180 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:32:38.922799 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:32:38.922764 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:32:55.618701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:32:55.618622 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:33:05.105944 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:33:05.105907 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:33:16.611485 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:33:16.611441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:33:28.106875 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:33:28.106822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:33:39.921223 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:33:39.921191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:33:50.906979 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:33:50.906944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:34:03.205198 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:03.205162 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:34:12.907108 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:12.907070 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:34:34.614928 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:34.614892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:34:46.700240 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:46.700197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:34:56.146805 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:56.146771 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:34:56.147283 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:34:56.147269 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:35:10.508724 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:35:10.508687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:35:21.803379 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:35:21.803341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:35:37.997349 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:35:37.997314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:35:47.799482 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:35:47.799393 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:35:57.303619 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:35:57.303568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:36:06.802725 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:06.802684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:36:16.706431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:16.706386 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:36:41.556898 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:41.556828 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:36:41.557528 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:41.557126 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" podUID="40614eb9-e1d2-4697-968d-e761abc73c0e" containerName="manager" containerID="cri-o://46c9380815cc6b5e0d5b020a8bc30187d9b7617c0abcd895f6ff08ce4b7c12ce" gracePeriod=10 Jul 02 20:36:44.897122 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:44.897088 2579 generic.go:356] "Generic (PLEG): container finished" podID="40614eb9-e1d2-4697-968d-e761abc73c0e" containerID="46c9380815cc6b5e0d5b020a8bc30187d9b7617c0abcd895f6ff08ce4b7c12ce" exitCode=0 Jul 02 20:36:44.897697 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:44.897158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" event={"ID":"40614eb9-e1d2-4697-968d-e761abc73c0e","Type":"ContainerDied","Data":"46c9380815cc6b5e0d5b020a8bc30187d9b7617c0abcd895f6ff08ce4b7c12ce"} Jul 02 20:36:44.917786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:44.917763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:36:45.040131 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.040031 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn5w4\" (UniqueName: \"kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4\") pod \"40614eb9-e1d2-4697-968d-e761abc73c0e\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " Jul 02 20:36:45.040316 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.040135 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume\") pod \"40614eb9-e1d2-4697-968d-e761abc73c0e\" (UID: \"40614eb9-e1d2-4697-968d-e761abc73c0e\") " Jul 02 20:36:45.040518 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.040493 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "40614eb9-e1d2-4697-968d-e761abc73c0e" (UID: "40614eb9-e1d2-4697-968d-e761abc73c0e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:36:45.042192 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.042172 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4" (OuterVolumeSpecName: "kube-api-access-kn5w4") pod "40614eb9-e1d2-4697-968d-e761abc73c0e" (UID: "40614eb9-e1d2-4697-968d-e761abc73c0e"). InnerVolumeSpecName "kube-api-access-kn5w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:36:45.140912 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.140878 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn5w4\" (UniqueName: \"kubernetes.io/projected/40614eb9-e1d2-4697-968d-e761abc73c0e-kube-api-access-kn5w4\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:36:45.140912 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.140909 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40614eb9-e1d2-4697-968d-e761abc73c0e-extensions-socket-volume\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:36:45.903359 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.903330 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" Jul 02 20:36:45.903861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.903362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w" event={"ID":"40614eb9-e1d2-4697-968d-e761abc73c0e","Type":"ContainerDied","Data":"e8a65257ad387b222e6b62aa440982cf411123e53567cfd170e56a045dfec427"} Jul 02 20:36:45.903861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.903415 2579 scope.go:117] "RemoveContainer" containerID="46c9380815cc6b5e0d5b020a8bc30187d9b7617c0abcd895f6ff08ce4b7c12ce" Jul 02 20:36:45.926073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.926049 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:36:45.929834 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:45.929805 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tc78w"] Jul 02 20:36:46.340264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:36:46.340180 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40614eb9-e1d2-4697-968d-e761abc73c0e" path="/var/lib/kubelet/pods/40614eb9-e1d2-4697-968d-e761abc73c0e/volumes" Jul 02 20:37:42.827805 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.827766 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf"] Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828150 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40614eb9-e1d2-4697-968d-e761abc73c0e" containerName="manager" Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828162 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40614eb9-e1d2-4697-968d-e761abc73c0e" containerName="manager" Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828174 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" containerName="authorino" Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828179 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" containerName="authorino" Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828234 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40614eb9-e1d2-4697-968d-e761abc73c0e" containerName="manager" Jul 02 20:37:42.828299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.828242 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc7ab3a1-3965-4aee-b5a9-7e6e0f5f3aba" containerName="authorino" Jul 02 20:37:42.831364 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.831347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:42.835514 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.835493 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6v7zk\"" Jul 02 20:37:42.858748 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.858717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf"] Jul 02 20:37:42.937653 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.937614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e95eb86-62a3-4347-957c-d068324317de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:42.937823 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:42.937688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfbv\" (UniqueName: \"kubernetes.io/projected/2e95eb86-62a3-4347-957c-d068324317de-kube-api-access-mcfbv\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.038387 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.038350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfbv\" (UniqueName: \"kubernetes.io/projected/2e95eb86-62a3-4347-957c-d068324317de-kube-api-access-mcfbv\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.038579 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.038445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e95eb86-62a3-4347-957c-d068324317de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.038910 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.038885 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e95eb86-62a3-4347-957c-d068324317de-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.057795 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.057766 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfbv\" (UniqueName: \"kubernetes.io/projected/2e95eb86-62a3-4347-957c-d068324317de-kube-api-access-mcfbv\") pod \"kuadrant-operator-controller-manager-86878c769f-ncrdf\" (UID: \"2e95eb86-62a3-4347-957c-d068324317de\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.141071 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.141040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:43.281182 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:43.280891 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf"] Jul 02 20:37:43.284231 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:37:43.284200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e95eb86_62a3_4347_957c_d068324317de.slice/crio-678084bbbadf2f61b7d0c7c4b002fd9bcffac7047978df10dbc007b3adde03eb WatchSource:0}: Error finding container 678084bbbadf2f61b7d0c7c4b002fd9bcffac7047978df10dbc007b3adde03eb: Status 404 returned error can't find the container with id 678084bbbadf2f61b7d0c7c4b002fd9bcffac7047978df10dbc007b3adde03eb Jul 02 20:37:44.173051 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:44.173010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" event={"ID":"2e95eb86-62a3-4347-957c-d068324317de","Type":"ContainerStarted","Data":"faa36a6bca9ce09b911f7a95d4f9866a751e20327e5e6c2f152388b32c155a8e"} Jul 02 20:37:44.173473 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:44.173076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" event={"ID":"2e95eb86-62a3-4347-957c-d068324317de","Type":"ContainerStarted","Data":"678084bbbadf2f61b7d0c7c4b002fd9bcffac7047978df10dbc007b3adde03eb"} Jul 02 20:37:44.173473 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:44.173193 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:37:44.203513 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:44.203460 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" podStartSLOduration=2.203443644 podStartE2EDuration="2.203443644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:37:44.202016538 +0000 UTC m=+1372.407403177" watchObservedRunningTime="2026-07-02 20:37:44.203443644 +0000 UTC m=+1372.408830273" Jul 02 20:37:55.180017 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:37:55.179982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ncrdf" Jul 02 20:38:03.711588 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:38:03.711547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:38:13.227784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:38:13.227748 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:38:28.128783 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:38:28.128742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:38:40.016598 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:38:40.016465 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:38:52.314902 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:38:52.314793 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:03.115452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:03.115414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:11.624144 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:11.624106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:23.212242 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:23.212206 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:34.326824 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:34.326792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:42.424661 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:42.424627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:53.413212 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:53.413180 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:39:56.184140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:56.184113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:39:56.186049 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:39:56.186027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:40:15.311948 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:40:15.311908 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:40:24.960285 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:40:24.960247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:40:34.912562 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:40:34.912521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:40:44.908466 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:40:44.908420 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:02.810079 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:02.810037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:13.095750 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:13.095713 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:22.809775 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:22.809741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:33.400494 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:33.400451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:43.005003 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:43.004911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:41:54.106968 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:41:54.106920 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:42:05.820925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:42:05.820886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:42:22.355700 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:42:22.355668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:42:34.808407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:42:34.808360 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:42:47.495168 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:42:47.495130 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:42:55.211386 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:42:55.211342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:43:08.208335 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:43:08.208294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:43:20.539472 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:43:20.539435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:43:33.701600 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:43:33.701558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:43:47.207290 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:43:47.207249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:43:56.307648 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:43:56.307612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:06.904252 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:06.904218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:07.877646 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.877611 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:07.881278 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.881256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:07.884907 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.884888 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-7ce02f69-serving-cert\"" Jul 02 20:44:07.886247 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.886227 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:44:07.905286 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.905259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:07.937400 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.937374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:07.937567 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.937421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t546k\" (UniqueName: \"kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:07.937567 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:07.937509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.038597 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.038562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.038784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.038609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t546k\" (UniqueName: \"kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.038784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.038650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.039248 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.039227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.041124 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.041103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.054047 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.054017 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t546k\" (UniqueName: \"kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k\") pod \"maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.191432 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.191343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:08.344800 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:44:08.344095 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388af253_b86e_4faf_84e4_1c423ea68366.slice/crio-7948ed779d8d20c2e511f2029893a8a6bd1258bcb68f44fb6636b85b85cb6735 WatchSource:0}: Error finding container 7948ed779d8d20c2e511f2029893a8a6bd1258bcb68f44fb6636b85b85cb6735: Status 404 returned error can't find the container with id 7948ed779d8d20c2e511f2029893a8a6bd1258bcb68f44fb6636b85b85cb6735 Jul 02 20:44:08.346367 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.346343 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:44:08.348300 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:08.348279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:09.017030 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:09.016991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" event={"ID":"388af253-b86e-4faf-84e4-1c423ea68366","Type":"ContainerStarted","Data":"7948ed779d8d20c2e511f2029893a8a6bd1258bcb68f44fb6636b85b85cb6735"} Jul 02 20:44:12.036090 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:12.036051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" event={"ID":"388af253-b86e-4faf-84e4-1c423ea68366","Type":"ContainerStarted","Data":"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0"} Jul 02 20:44:12.036575 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:12.036170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:12.054200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:12.054096 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" podStartSLOduration=2.303104764 podStartE2EDuration="5.054079695s" totalImagesPullingTime="2.750974931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:44:12.053687917 +0000 UTC m=+1760.259074546" watchObservedRunningTime="2026-07-02 20:44:12.054079695 +0000 UTC m=+1760.259466330" Jul 02 20:44:12.843392 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:12.843361 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:13.187941 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:13.187904 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-7ce02f69-serving-cert: secret "maas-api-e2e-ait-7ce02f69-serving-cert" not found Jul 02 20:44:13.188325 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:13.187983 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls podName:388af253-b86e-4faf-84e4-1c423ea68366 nodeName:}" failed. No retries permitted until 2026-07-02 20:44:13.687965843 +0000 UTC m=+1761.893352451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls") pod "maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" (UID: "388af253-b86e-4faf-84e4-1c423ea68366") : secret "maas-api-e2e-ait-7ce02f69-serving-cert" not found Jul 02 20:44:13.693119 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:13.693080 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-7ce02f69-serving-cert: secret "maas-api-e2e-ait-7ce02f69-serving-cert" not found Jul 02 20:44:13.693311 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:13.693150 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls podName:388af253-b86e-4faf-84e4-1c423ea68366 nodeName:}" failed. No retries permitted until 2026-07-02 20:44:14.693135674 +0000 UTC m=+1762.898522280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls") pod "maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" (UID: "388af253-b86e-4faf-84e4-1c423ea68366") : secret "maas-api-e2e-ait-7ce02f69-serving-cert" not found Jul 02 20:44:14.053228 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.053118 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" podUID="388af253-b86e-4faf-84e4-1c423ea68366" containerName="maas-api" containerID="cri-o://14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0" gracePeriod=30 Jul 02 20:44:14.304027 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.303953 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:14.400675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.400646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls\") pod \"388af253-b86e-4faf-84e4-1c423ea68366\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " Jul 02 20:44:14.400906 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.400762 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca\") pod \"388af253-b86e-4faf-84e4-1c423ea68366\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " Jul 02 20:44:14.400906 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.400804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t546k\" (UniqueName: \"kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k\") pod \"388af253-b86e-4faf-84e4-1c423ea68366\" (UID: \"388af253-b86e-4faf-84e4-1c423ea68366\") " Jul 02 20:44:14.401229 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.401198 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "388af253-b86e-4faf-84e4-1c423ea68366" (UID: "388af253-b86e-4faf-84e4-1c423ea68366"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:44:14.403174 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.403148 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k" (OuterVolumeSpecName: "kube-api-access-t546k") pod "388af253-b86e-4faf-84e4-1c423ea68366" (UID: "388af253-b86e-4faf-84e4-1c423ea68366"). InnerVolumeSpecName "kube-api-access-t546k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:44:14.403264 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.403230 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "388af253-b86e-4faf-84e4-1c423ea68366" (UID: "388af253-b86e-4faf-84e4-1c423ea68366"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:44:14.501742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.501704 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388af253-b86e-4faf-84e4-1c423ea68366-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:14.501742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.501733 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t546k\" (UniqueName: \"kubernetes.io/projected/388af253-b86e-4faf-84e4-1c423ea68366-kube-api-access-t546k\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:14.501742 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:14.501743 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388af253-b86e-4faf-84e4-1c423ea68366-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:15.059534 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.059499 2579 generic.go:356] "Generic (PLEG): container finished" podID="388af253-b86e-4faf-84e4-1c423ea68366" containerID="14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0" exitCode=0 Jul 02 20:44:15.059702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.059540 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" Jul 02 20:44:15.059702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.059577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" event={"ID":"388af253-b86e-4faf-84e4-1c423ea68366","Type":"ContainerDied","Data":"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0"} Jul 02 20:44:15.059702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.059623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t" event={"ID":"388af253-b86e-4faf-84e4-1c423ea68366","Type":"ContainerDied","Data":"7948ed779d8d20c2e511f2029893a8a6bd1258bcb68f44fb6636b85b85cb6735"} Jul 02 20:44:15.059702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.059645 2579 scope.go:117] "RemoveContainer" containerID="14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0" Jul 02 20:44:15.068351 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.068330 2579 scope.go:117] "RemoveContainer" containerID="14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0" Jul 02 20:44:15.068609 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:15.068590 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0\": container with ID starting with 14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0 not found: ID does not exist" containerID="14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0" Jul 02 20:44:15.068681 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.068617 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0"} err="failed to get container status \"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0\": rpc error: code = NotFound desc = could not find container \"14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0\": container with ID starting with 14b77a202887955da24be3db900175ca0b676bc08f5ecabad7f30bf0ad327fe0 not found: ID does not exist" Jul 02 20:44:15.080327 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.080300 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:15.083884 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:15.083862 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-7ce02f69-6bf8d6974c-42x5t"] Jul 02 20:44:16.339661 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:16.339623 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388af253-b86e-4faf-84e4-1c423ea68366" path="/var/lib/kubelet/pods/388af253-b86e-4faf-84e4-1c423ea68366/volumes" Jul 02 20:44:23.202788 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:23.202749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:33.600727 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:33.600692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:39.161569 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.161536 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:39.161991 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.161938 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388af253-b86e-4faf-84e4-1c423ea68366" containerName="maas-api" Jul 02 20:44:39.161991 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.161952 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="388af253-b86e-4faf-84e4-1c423ea68366" containerName="maas-api" Jul 02 20:44:39.162066 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.162006 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="388af253-b86e-4faf-84e4-1c423ea68366" containerName="maas-api" Jul 02 20:44:39.165282 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.165263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.167993 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.167967 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:44:39.168144 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.168089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-3986647f-serving-cert\"" Jul 02 20:44:39.178215 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.178161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:39.238508 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.238469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.238508 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.238507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.238721 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.238604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5q9\" (UniqueName: \"kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.339711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.339669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.339711 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.339716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.340041 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.339798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5q9\" (UniqueName: \"kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.340041 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:39.339951 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-3986647f-serving-cert: secret "maas-api-e2e-derive-3986647f-serving-cert" not found Jul 02 20:44:39.340153 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:39.340060 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls podName:795aac29-a607-425c-8a4f-7525f94fbaa0 nodeName:}" failed. No retries permitted until 2026-07-02 20:44:39.840035472 +0000 UTC m=+1788.045422090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls") pod "maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" (UID: "795aac29-a607-425c-8a4f-7525f94fbaa0") : secret "maas-api-e2e-derive-3986647f-serving-cert" not found Jul 02 20:44:39.340415 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.340392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.359347 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.359317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5q9\" (UniqueName: \"kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.844632 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.844565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:39.847169 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:39.847140 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") pod \"maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:40.078012 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:40.077968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:40.221360 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:40.221331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:40.223485 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:44:40.223453 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795aac29_a607_425c_8a4f_7525f94fbaa0.slice/crio-6a771ee2eaa2d6a985d9d3b6ce739791342ec9562892e31814b8d47943325a7c WatchSource:0}: Error finding container 6a771ee2eaa2d6a985d9d3b6ce739791342ec9562892e31814b8d47943325a7c: Status 404 returned error can't find the container with id 6a771ee2eaa2d6a985d9d3b6ce739791342ec9562892e31814b8d47943325a7c Jul 02 20:44:41.196598 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:41.196553 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" event={"ID":"795aac29-a607-425c-8a4f-7525f94fbaa0","Type":"ContainerStarted","Data":"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309"} Jul 02 20:44:41.196785 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:41.196620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" event={"ID":"795aac29-a607-425c-8a4f-7525f94fbaa0","Type":"ContainerStarted","Data":"6a771ee2eaa2d6a985d9d3b6ce739791342ec9562892e31814b8d47943325a7c"} Jul 02 20:44:41.196785 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:41.196743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:41.216231 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:41.216183 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" podStartSLOduration=1.8376818529999999 podStartE2EDuration="2.216167796s" totalImagesPullingTime="378.485943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:44:41.21495252 +0000 UTC m=+1789.420339168" watchObservedRunningTime="2026-07-02 20:44:41.216167796 +0000 UTC m=+1789.421554425" Jul 02 20:44:43.104322 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:43.104228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:43.809744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:43.809711 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:43.809972 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:43.809950 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" podUID="795aac29-a607-425c-8a4f-7525f94fbaa0" containerName="maas-api" containerID="cri-o://f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309" gracePeriod=30 Jul 02 20:44:44.067121 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.067053 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:44.082420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.082397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") pod \"795aac29-a607-425c-8a4f-7525f94fbaa0\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " Jul 02 20:44:44.082553 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.082428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl5q9\" (UniqueName: \"kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9\") pod \"795aac29-a607-425c-8a4f-7525f94fbaa0\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " Jul 02 20:44:44.082600 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.082553 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca\") pod \"795aac29-a607-425c-8a4f-7525f94fbaa0\" (UID: \"795aac29-a607-425c-8a4f-7525f94fbaa0\") " Jul 02 20:44:44.082900 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.082880 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "795aac29-a607-425c-8a4f-7525f94fbaa0" (UID: "795aac29-a607-425c-8a4f-7525f94fbaa0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:44:44.084875 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.084815 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "795aac29-a607-425c-8a4f-7525f94fbaa0" (UID: "795aac29-a607-425c-8a4f-7525f94fbaa0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:44:44.085015 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.084886 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9" (OuterVolumeSpecName: "kube-api-access-dl5q9") pod "795aac29-a607-425c-8a4f-7525f94fbaa0" (UID: "795aac29-a607-425c-8a4f-7525f94fbaa0"). InnerVolumeSpecName "kube-api-access-dl5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:44:44.183685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.183654 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/795aac29-a607-425c-8a4f-7525f94fbaa0-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:44.183685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.183682 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/795aac29-a607-425c-8a4f-7525f94fbaa0-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:44.183685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.183692 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl5q9\" (UniqueName: \"kubernetes.io/projected/795aac29-a607-425c-8a4f-7525f94fbaa0-kube-api-access-dl5q9\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:44:44.210076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.210043 2579 generic.go:356] "Generic (PLEG): container finished" podID="795aac29-a607-425c-8a4f-7525f94fbaa0" containerID="f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309" exitCode=0 Jul 02 20:44:44.210256 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.210091 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" Jul 02 20:44:44.210256 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.210131 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" event={"ID":"795aac29-a607-425c-8a4f-7525f94fbaa0","Type":"ContainerDied","Data":"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309"} Jul 02 20:44:44.210256 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.210190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq" event={"ID":"795aac29-a607-425c-8a4f-7525f94fbaa0","Type":"ContainerDied","Data":"6a771ee2eaa2d6a985d9d3b6ce739791342ec9562892e31814b8d47943325a7c"} Jul 02 20:44:44.210256 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.210215 2579 scope.go:117] "RemoveContainer" containerID="f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309" Jul 02 20:44:44.219308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.219291 2579 scope.go:117] "RemoveContainer" containerID="f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309" Jul 02 20:44:44.219575 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:44:44.219554 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309\": container with ID starting with f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309 not found: ID does not exist" containerID="f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309" Jul 02 20:44:44.219641 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.219583 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309"} err="failed to get container status \"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309\": rpc error: code = NotFound desc = could not find container \"f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309\": container with ID starting with f9e172595c6851b546b9375a04ce2aa7217b534e457452e548222845833f2309 not found: ID does not exist" Jul 02 20:44:44.232670 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.232647 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:44.239706 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.239681 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-3986647f-59c9c7566c-8xqvq"] Jul 02 20:44:44.340889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:44.340795 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795aac29-a607-425c-8a4f-7525f94fbaa0" path="/var/lib/kubelet/pods/795aac29-a607-425c-8a4f-7525f94fbaa0/volumes" Jul 02 20:44:54.104481 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:54.104442 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:44:56.215276 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:56.215249 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:44:56.218720 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:44:56.218695 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:46:12.611294 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:12.611206 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:46:22.118045 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:22.118009 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:46:29.691922 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.691888 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:29.692416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.692243 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795aac29-a607-425c-8a4f-7525f94fbaa0" containerName="maas-api" Jul 02 20:46:29.692416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.692254 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="795aac29-a607-425c-8a4f-7525f94fbaa0" containerName="maas-api" Jul 02 20:46:29.692416 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.692330 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="795aac29-a607-425c-8a4f-7525f94fbaa0" containerName="maas-api" Jul 02 20:46:29.695245 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.695226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.697956 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.697935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-5b080c83-serving-cert\"" Jul 02 20:46:29.698150 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.698132 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:46:29.710928 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.709597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:29.725477 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.725441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmpc\" (UniqueName: \"kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.725623 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.725582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.725623 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.725613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.826151 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.826112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.826340 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.826159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.826340 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.826227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmpc\" (UniqueName: \"kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.826710 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:46:29.826341 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-5b080c83-serving-cert: secret "maas-api-e2e-mt-5b080c83-serving-cert" not found Jul 02 20:46:29.826710 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:46:29.826424 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls podName:863c88ef-93bf-482c-bd63-312b477d9100 nodeName:}" failed. No retries permitted until 2026-07-02 20:46:30.326400043 +0000 UTC m=+1898.531786670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls") pod "maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" (UID: "863c88ef-93bf-482c-bd63-312b477d9100") : secret "maas-api-e2e-mt-5b080c83-serving-cert" not found Jul 02 20:46:29.826877 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.826829 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:29.837338 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:29.837317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmpc\" (UniqueName: \"kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:30.331051 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:30.331006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:30.333560 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:30.333534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") pod \"maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:30.605674 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:30.605577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:30.748577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:30.748551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:30.752162 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:46:30.752129 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863c88ef_93bf_482c_bd63_312b477d9100.slice/crio-dbaa8a18597222b7ed8bda42222ae4ba81d59b325fe87ad054b7557d0e26d86d WatchSource:0}: Error finding container dbaa8a18597222b7ed8bda42222ae4ba81d59b325fe87ad054b7557d0e26d86d: Status 404 returned error can't find the container with id dbaa8a18597222b7ed8bda42222ae4ba81d59b325fe87ad054b7557d0e26d86d Jul 02 20:46:31.727969 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:31.727935 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" event={"ID":"863c88ef-93bf-482c-bd63-312b477d9100","Type":"ContainerStarted","Data":"8b9dd9329a187222271cfd473fadabea836202b99404121eceef8d30c27f733f"} Jul 02 20:46:31.728167 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:31.727988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" event={"ID":"863c88ef-93bf-482c-bd63-312b477d9100","Type":"ContainerStarted","Data":"dbaa8a18597222b7ed8bda42222ae4ba81d59b325fe87ad054b7557d0e26d86d"} Jul 02 20:46:31.728167 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:31.728028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:31.746140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:31.746079 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" podStartSLOduration=2.208844789 podStartE2EDuration="2.746057279s" totalImagesPullingTime="537.21249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:46:31.743439942 +0000 UTC m=+1899.948826595" watchObservedRunningTime="2026-07-02 20:46:31.746057279 +0000 UTC m=+1899.951443910" Jul 02 20:46:34.210864 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:34.210804 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:46:37.738106 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:37.738075 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:41.654374 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:41.654291 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:41.654871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:41.654528 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" podUID="863c88ef-93bf-482c-bd63-312b477d9100" containerName="maas-api" containerID="cri-o://8b9dd9329a187222271cfd473fadabea836202b99404121eceef8d30c27f733f" gracePeriod=30 Jul 02 20:46:41.778278 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:41.778241 2579 generic.go:356] "Generic (PLEG): container finished" podID="863c88ef-93bf-482c-bd63-312b477d9100" containerID="8b9dd9329a187222271cfd473fadabea836202b99404121eceef8d30c27f733f" exitCode=0 Jul 02 20:46:41.778429 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:41.778313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" event={"ID":"863c88ef-93bf-482c-bd63-312b477d9100","Type":"ContainerDied","Data":"8b9dd9329a187222271cfd473fadabea836202b99404121eceef8d30c27f733f"} Jul 02 20:46:41.912670 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:41.912613 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:42.041897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.041861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca\") pod \"863c88ef-93bf-482c-bd63-312b477d9100\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " Jul 02 20:46:42.042100 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.041918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdmpc\" (UniqueName: \"kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc\") pod \"863c88ef-93bf-482c-bd63-312b477d9100\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " Jul 02 20:46:42.042100 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.041950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") pod \"863c88ef-93bf-482c-bd63-312b477d9100\" (UID: \"863c88ef-93bf-482c-bd63-312b477d9100\") " Jul 02 20:46:42.042307 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.042280 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "863c88ef-93bf-482c-bd63-312b477d9100" (UID: "863c88ef-93bf-482c-bd63-312b477d9100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:46:42.044404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.044377 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "863c88ef-93bf-482c-bd63-312b477d9100" (UID: "863c88ef-93bf-482c-bd63-312b477d9100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:46:42.044527 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.044413 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc" (OuterVolumeSpecName: "kube-api-access-rdmpc") pod "863c88ef-93bf-482c-bd63-312b477d9100" (UID: "863c88ef-93bf-482c-bd63-312b477d9100"). InnerVolumeSpecName "kube-api-access-rdmpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:46:42.143464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.143414 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c88ef-93bf-482c-bd63-312b477d9100-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:46:42.143464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.143457 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmpc\" (UniqueName: \"kubernetes.io/projected/863c88ef-93bf-482c-bd63-312b477d9100-kube-api-access-rdmpc\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:46:42.143464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.143473 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c88ef-93bf-482c-bd63-312b477d9100-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:46:42.784177 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.784141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" event={"ID":"863c88ef-93bf-482c-bd63-312b477d9100","Type":"ContainerDied","Data":"dbaa8a18597222b7ed8bda42222ae4ba81d59b325fe87ad054b7557d0e26d86d"} Jul 02 20:46:42.784177 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.784163 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn" Jul 02 20:46:42.784785 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.784205 2579 scope.go:117] "RemoveContainer" containerID="8b9dd9329a187222271cfd473fadabea836202b99404121eceef8d30c27f733f" Jul 02 20:46:42.801747 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.801711 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:42.807825 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:42.807788 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-5b080c83-564df7dd47-9pwmn"] Jul 02 20:46:44.339853 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:44.339811 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863c88ef-93bf-482c-bd63-312b477d9100" path="/var/lib/kubelet/pods/863c88ef-93bf-482c-bd63-312b477d9100/volumes" Jul 02 20:46:45.706103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:46:45.706066 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:47:09.750866 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.750821 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:09.751236 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.751222 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863c88ef-93bf-482c-bd63-312b477d9100" containerName="maas-api" Jul 02 20:47:09.751294 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.751238 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="863c88ef-93bf-482c-bd63-312b477d9100" containerName="maas-api" Jul 02 20:47:09.751353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.751321 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="863c88ef-93bf-482c-bd63-312b477d9100" containerName="maas-api" Jul 02 20:47:09.754221 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.754203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.757073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.757052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:47:09.757073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.757064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-13937e3a-serving-cert\"" Jul 02 20:47:09.766067 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.766043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:09.898969 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.898925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgx8\" (UniqueName: \"kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.898969 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.898978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.899194 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.899009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.999490 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.999452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.999688 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.999539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgx8\" (UniqueName: \"kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:09.999688 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:09.999566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.000242 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.000220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.002083 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.002029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.008399 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.008375 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgx8\" (UniqueName: \"kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8\") pod \"maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.065701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.065658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.210345 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.210323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:10.212491 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:47:10.212462 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844ea0db_c865_427f_96b5_8dd89b1c2e09.slice/crio-a0f741ca94e45af89e0c2beecac787954b6bf7aa51369fea0977c7cad96d3d78 WatchSource:0}: Error finding container a0f741ca94e45af89e0c2beecac787954b6bf7aa51369fea0977c7cad96d3d78: Status 404 returned error can't find the container with id a0f741ca94e45af89e0c2beecac787954b6bf7aa51369fea0977c7cad96d3d78 Jul 02 20:47:10.922905 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.922865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" event={"ID":"844ea0db-c865-427f-96b5-8dd89b1c2e09","Type":"ContainerStarted","Data":"b8c6c3ae637606ec63bd17bf93a5d48effbd10f5ff7b72d478a6dd3f16a9c844"} Jul 02 20:47:10.923369 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.922917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" event={"ID":"844ea0db-c865-427f-96b5-8dd89b1c2e09","Type":"ContainerStarted","Data":"a0f741ca94e45af89e0c2beecac787954b6bf7aa51369fea0977c7cad96d3d78"} Jul 02 20:47:10.923369 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.922947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:10.941098 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:10.941034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" podStartSLOduration=1.616983345 podStartE2EDuration="1.941019361s" totalImagesPullingTime="324.036016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:47:10.940571075 +0000 UTC m=+1939.145957722" watchObservedRunningTime="2026-07-02 20:47:10.941019361 +0000 UTC m=+1939.146405990" Jul 02 20:47:12.303428 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:12.303395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:47:16.932226 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:16.932199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:29.504392 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:29.504355 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:47:44.314274 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:44.314173 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:47:46.930517 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:46.930487 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:46.930979 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:46.930701 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" podUID="844ea0db-c865-427f-96b5-8dd89b1c2e09" containerName="maas-api" containerID="cri-o://b8c6c3ae637606ec63bd17bf93a5d48effbd10f5ff7b72d478a6dd3f16a9c844" gracePeriod=30 Jul 02 20:47:47.109760 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.109721 2579 generic.go:356] "Generic (PLEG): container finished" podID="844ea0db-c865-427f-96b5-8dd89b1c2e09" containerID="b8c6c3ae637606ec63bd17bf93a5d48effbd10f5ff7b72d478a6dd3f16a9c844" exitCode=0 Jul 02 20:47:47.109915 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.109796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" event={"ID":"844ea0db-c865-427f-96b5-8dd89b1c2e09","Type":"ContainerDied","Data":"b8c6c3ae637606ec63bd17bf93a5d48effbd10f5ff7b72d478a6dd3f16a9c844"} Jul 02 20:47:47.182724 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.182663 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:47.340292 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.340266 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgx8\" (UniqueName: \"kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8\") pod \"844ea0db-c865-427f-96b5-8dd89b1c2e09\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " Jul 02 20:47:47.340475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.340375 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls\") pod \"844ea0db-c865-427f-96b5-8dd89b1c2e09\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " Jul 02 20:47:47.340475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.340405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca\") pod \"844ea0db-c865-427f-96b5-8dd89b1c2e09\" (UID: \"844ea0db-c865-427f-96b5-8dd89b1c2e09\") " Jul 02 20:47:47.340909 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.340879 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "844ea0db-c865-427f-96b5-8dd89b1c2e09" (UID: "844ea0db-c865-427f-96b5-8dd89b1c2e09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:47:47.342544 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.342518 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8" (OuterVolumeSpecName: "kube-api-access-hcgx8") pod "844ea0db-c865-427f-96b5-8dd89b1c2e09" (UID: "844ea0db-c865-427f-96b5-8dd89b1c2e09"). InnerVolumeSpecName "kube-api-access-hcgx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:47:47.342624 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.342548 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "844ea0db-c865-427f-96b5-8dd89b1c2e09" (UID: "844ea0db-c865-427f-96b5-8dd89b1c2e09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:47:47.442621 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.442518 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgx8\" (UniqueName: \"kubernetes.io/projected/844ea0db-c865-427f-96b5-8dd89b1c2e09-kube-api-access-hcgx8\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:47:47.442621 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.442554 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/844ea0db-c865-427f-96b5-8dd89b1c2e09-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:47:47.442621 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:47.442568 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/844ea0db-c865-427f-96b5-8dd89b1c2e09-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:47:48.116379 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.116348 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" Jul 02 20:47:48.116889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.116383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz" event={"ID":"844ea0db-c865-427f-96b5-8dd89b1c2e09","Type":"ContainerDied","Data":"a0f741ca94e45af89e0c2beecac787954b6bf7aa51369fea0977c7cad96d3d78"} Jul 02 20:47:48.116889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.116438 2579 scope.go:117] "RemoveContainer" containerID="b8c6c3ae637606ec63bd17bf93a5d48effbd10f5ff7b72d478a6dd3f16a9c844" Jul 02 20:47:48.138047 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.138014 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:48.140407 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.140386 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-13937e3a-fcfdf77bb-r68jz"] Jul 02 20:47:48.340093 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:48.340063 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="844ea0db-c865-427f-96b5-8dd89b1c2e09" path="/var/lib/kubelet/pods/844ea0db-c865-427f-96b5-8dd89b1c2e09/volumes" Jul 02 20:47:55.206561 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:47:55.206528 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:48:36.903662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:48:36.903629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:48:46.703636 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:48:46.703595 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:48:58.805145 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:48:58.805102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:49:39.620544 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.620513 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:49:39.620970 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.620902 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="844ea0db-c865-427f-96b5-8dd89b1c2e09" containerName="maas-api" Jul 02 20:49:39.620970 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.620914 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="844ea0db-c865-427f-96b5-8dd89b1c2e09" containerName="maas-api" Jul 02 20:49:39.621051 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.620987 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="844ea0db-c865-427f-96b5-8dd89b1c2e09" containerName="maas-api" Jul 02 20:49:39.623631 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.623614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.626740 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.626718 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:49:39.626949 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.626933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-587503-serving-cert\"" Jul 02 20:49:39.633768 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.633746 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:49:39.645290 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.645258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.645472 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.645358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.645472 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.645387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.746001 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.745949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.746213 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.746030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.746213 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.746053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.746213 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:49:39.746117 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-587503-serving-cert: secret "maas-api-e2e-inf-a-587503-serving-cert" not found Jul 02 20:49:39.746213 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:49:39.746194 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls podName:d8a4be9e-bfa5-4145-8326-c17f33ca70d7 nodeName:}" failed. No retries permitted until 2026-07-02 20:49:40.246173273 +0000 UTC m=+2088.451559882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls") pod "maas-api-e2e-inf-a-587503-67f65757f4-2kl76" (UID: "d8a4be9e-bfa5-4145-8326-c17f33ca70d7") : secret "maas-api-e2e-inf-a-587503-serving-cert" not found Jul 02 20:49:39.746858 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.746818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:39.758602 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:39.758580 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:40.250730 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:40.250689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:40.253457 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:40.253424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") pod \"maas-api-e2e-inf-a-587503-67f65757f4-2kl76\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:40.534036 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:40.533940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:40.673568 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:40.673540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:49:40.675669 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:49:40.675638 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a4be9e_bfa5_4145_8326_c17f33ca70d7.slice/crio-0b0352ffb9808676464832e0291caa6c466c7e77056a3748a03bb1887e30cbae WatchSource:0}: Error finding container 0b0352ffb9808676464832e0291caa6c466c7e77056a3748a03bb1887e30cbae: Status 404 returned error can't find the container with id 0b0352ffb9808676464832e0291caa6c466c7e77056a3748a03bb1887e30cbae Jul 02 20:49:40.676816 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:40.676799 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:49:41.656860 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:41.656754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" event={"ID":"d8a4be9e-bfa5-4145-8326-c17f33ca70d7","Type":"ContainerStarted","Data":"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1"} Jul 02 20:49:41.656860 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:41.656814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" event={"ID":"d8a4be9e-bfa5-4145-8326-c17f33ca70d7","Type":"ContainerStarted","Data":"0b0352ffb9808676464832e0291caa6c466c7e77056a3748a03bb1887e30cbae"} Jul 02 20:49:41.657078 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:41.656873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:41.674545 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:41.674486 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" podStartSLOduration=1.970536945 podStartE2EDuration="2.674464903s" totalImagesPullingTime="703.927958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:49:41.673994035 +0000 UTC m=+2089.879380684" watchObservedRunningTime="2026-07-02 20:49:41.674464903 +0000 UTC m=+2089.879851532" Jul 02 20:49:47.099023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.098985 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:49:47.666975 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.666941 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:49:47.757361 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.757327 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:49:47.761104 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.761087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.765436 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.765415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-237f0d-serving-cert\"" Jul 02 20:49:47.786187 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.786153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:49:47.821675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.821638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7np\" (UniqueName: \"kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.821899 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.821732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.821899 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.821811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.922489 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.922402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.922489 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.922462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.922716 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.922484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7np\" (UniqueName: \"kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.922716 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:49:47.922549 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-237f0d-serving-cert: secret "maas-api-e2e-inf-b-237f0d-serving-cert" not found Jul 02 20:49:47.922716 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:49:47.922609 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls podName:c2166837-940f-4366-8f6d-fa3696481d01 nodeName:}" failed. No retries permitted until 2026-07-02 20:49:48.422593528 +0000 UTC m=+2096.627980135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls") pod "maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" (UID: "c2166837-940f-4366-8f6d-fa3696481d01") : secret "maas-api-e2e-inf-b-237f0d-serving-cert" not found Jul 02 20:49:47.923195 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.923173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:47.931401 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:47.931377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7np\" (UniqueName: \"kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:48.427578 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:48.427540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:48.430136 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:48.430116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") pod \"maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:48.672375 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:48.672332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:48.812203 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:48.812177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:49:48.814670 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:49:48.814642 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2166837_940f_4366_8f6d_fa3696481d01.slice/crio-12a21263e152938fc5084ad132604090b61500cbc3f41ecab07de7c14488d4b6 WatchSource:0}: Error finding container 12a21263e152938fc5084ad132604090b61500cbc3f41ecab07de7c14488d4b6: Status 404 returned error can't find the container with id 12a21263e152938fc5084ad132604090b61500cbc3f41ecab07de7c14488d4b6 Jul 02 20:49:49.697634 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:49.697596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" event={"ID":"c2166837-940f-4366-8f6d-fa3696481d01","Type":"ContainerStarted","Data":"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168"} Jul 02 20:49:49.698091 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:49.697649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" event={"ID":"c2166837-940f-4366-8f6d-fa3696481d01","Type":"ContainerStarted","Data":"12a21263e152938fc5084ad132604090b61500cbc3f41ecab07de7c14488d4b6"} Jul 02 20:49:49.698091 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:49.697712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:49.715238 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:49.715186 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" podStartSLOduration=2.378080689 podStartE2EDuration="2.715168024s" totalImagesPullingTime="337.087335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:49:49.71298761 +0000 UTC m=+2097.918374250" watchObservedRunningTime="2026-07-02 20:49:49.715168024 +0000 UTC m=+2097.920554652" Jul 02 20:49:55.708345 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:55.708316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:49:56.245982 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:56.245955 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:49:56.251141 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:56.251115 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:49:58.404469 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:49:58.404428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:50:06.702938 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:06.702902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:50:44.311662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.311564 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:50:44.315749 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.315727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.318692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.318667 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-237f0d\"/\"openshift-service-ca.crt\"" Jul 02 20:50:44.319626 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.319607 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-237f0d\"/\"default-dockercfg-njwxt\"" Jul 02 20:50:44.319870 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.319850 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-237f0d\"/\"kube-root-ca.crt\"" Jul 02 20:50:44.319870 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.319860 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-237f0d\"/\"test-model-237f0d-kserve-self-signed-certs\"" Jul 02 20:50:44.329520 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.328488 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:50:44.431684 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.431908 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rfm\" (UniqueName: \"kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.431908 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.431908 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.432059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.432059 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.431954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.532889 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.532825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.532897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.532927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.532950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.533004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.533038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rfm\" (UniqueName: \"kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.533301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.533316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.533462 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.533433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.535345 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.535319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.535740 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.535717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.540997 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.540978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rfm\" (UniqueName: \"kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm\") pod \"test-model-237f0d-kserve-5c5645d65c-mbfzv\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.631266 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.631226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:44.770798 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.770772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:50:44.772692 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:50:44.772668 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ace2724_f45d_459b_8ce2_2ee7d9542035.slice/crio-c0b2a9b13e9209bc5d498ff845e83784897e5caa9c9e75a75d2466c2fd61b57e WatchSource:0}: Error finding container c0b2a9b13e9209bc5d498ff845e83784897e5caa9c9e75a75d2466c2fd61b57e: Status 404 returned error can't find the container with id c0b2a9b13e9209bc5d498ff845e83784897e5caa9c9e75a75d2466c2fd61b57e Jul 02 20:50:44.968667 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.968574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" event={"ID":"3ace2724-f45d-459b-8ce2-2ee7d9542035","Type":"ContainerStarted","Data":"2b227a44d0df983b9958e41bede91a60db50a5a6480093eff89be6538f364e7c"} Jul 02 20:50:44.968667 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.968619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" event={"ID":"3ace2724-f45d-459b-8ce2-2ee7d9542035","Type":"ContainerStarted","Data":"c0b2a9b13e9209bc5d498ff845e83784897e5caa9c9e75a75d2466c2fd61b57e"} Jul 02 20:50:44.985181 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:44.985130 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" podStartSLOduration=0.985116007 podStartE2EDuration="985.116007ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:50:44.984799533 +0000 UTC m=+2153.190186173" watchObservedRunningTime="2026-07-02 20:50:44.985116007 +0000 UTC m=+2153.190502636" Jul 02 20:50:48.944209 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:48.944173 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:50:54.631383 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:54.631351 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:54.631793 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:54.631403 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:54.643901 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:54.643877 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:50:55.030036 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:50:55.029943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:51:03.207993 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:03.207954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:51:06.990678 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:06.990643 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:51:07.799453 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:07.799426 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:51:07.799662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:07.799637 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" podUID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" containerName="maas-api" containerID="cri-o://fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1" gracePeriod=30 Jul 02 20:51:08.046477 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.046440 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:51:08.083526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.083424 2579 generic.go:356] "Generic (PLEG): container finished" podID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" containerID="fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1" exitCode=0 Jul 02 20:51:08.083526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.083472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" event={"ID":"d8a4be9e-bfa5-4145-8326-c17f33ca70d7","Type":"ContainerDied","Data":"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1"} Jul 02 20:51:08.083526 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.083488 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" Jul 02 20:51:08.083783 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.083528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76" event={"ID":"d8a4be9e-bfa5-4145-8326-c17f33ca70d7","Type":"ContainerDied","Data":"0b0352ffb9808676464832e0291caa6c466c7e77056a3748a03bb1887e30cbae"} Jul 02 20:51:08.083783 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.083559 2579 scope.go:117] "RemoveContainer" containerID="fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1" Jul 02 20:51:08.092451 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.092427 2579 scope.go:117] "RemoveContainer" containerID="fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1" Jul 02 20:51:08.092746 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:51:08.092718 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1\": container with ID starting with fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1 not found: ID does not exist" containerID="fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1" Jul 02 20:51:08.092902 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.092747 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1"} err="failed to get container status \"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1\": rpc error: code = NotFound desc = could not find container \"fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1\": container with ID starting with fd77e05ae64583658b459edca75bafb029e980f6f7fabade12d0c2f2c5742bf1 not found: ID does not exist" Jul 02 20:51:08.159141 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.159103 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca\") pod \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " Jul 02 20:51:08.159141 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.159145 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") pod \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " Jul 02 20:51:08.159395 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.159179 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555\") pod \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\" (UID: \"d8a4be9e-bfa5-4145-8326-c17f33ca70d7\") " Jul 02 20:51:08.159571 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.159548 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8a4be9e-bfa5-4145-8326-c17f33ca70d7" (UID: "d8a4be9e-bfa5-4145-8326-c17f33ca70d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:51:08.161452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.161425 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8a4be9e-bfa5-4145-8326-c17f33ca70d7" (UID: "d8a4be9e-bfa5-4145-8326-c17f33ca70d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:51:08.161555 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.161428 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555" (OuterVolumeSpecName: "kube-api-access-vj555") pod "d8a4be9e-bfa5-4145-8326-c17f33ca70d7" (UID: "d8a4be9e-bfa5-4145-8326-c17f33ca70d7"). InnerVolumeSpecName "kube-api-access-vj555". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:51:08.260534 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.260499 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj555\" (UniqueName: \"kubernetes.io/projected/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-kube-api-access-vj555\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:08.260534 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.260527 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:08.260534 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.260537 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8a4be9e-bfa5-4145-8326-c17f33ca70d7-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:08.400207 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.400175 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:51:08.401925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:08.401899 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-587503-67f65757f4-2kl76"] Jul 02 20:51:10.342918 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:10.342881 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" path="/var/lib/kubelet/pods/d8a4be9e-bfa5-4145-8326-c17f33ca70d7/volumes" Jul 02 20:51:19.802431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:19.802393 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:51:22.012834 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.012769 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" podUID="3ace2724-f45d-459b-8ce2-2ee7d9542035" containerName="main" containerID="cri-o://2b227a44d0df983b9958e41bede91a60db50a5a6480093eff89be6538f364e7c" gracePeriod=45 Jul 02 20:51:22.154349 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.154318 2579 generic.go:356] "Generic (PLEG): container finished" podID="3ace2724-f45d-459b-8ce2-2ee7d9542035" containerID="2b227a44d0df983b9958e41bede91a60db50a5a6480093eff89be6538f364e7c" exitCode=0 Jul 02 20:51:22.154552 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.154405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" event={"ID":"3ace2724-f45d-459b-8ce2-2ee7d9542035","Type":"ContainerDied","Data":"2b227a44d0df983b9958e41bede91a60db50a5a6480093eff89be6538f364e7c"} Jul 02 20:51:22.263995 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.263930 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:51:22.283712 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283682 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.283931 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7rfm\" (UniqueName: \"kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.283931 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283822 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.283931 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.283931 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.284156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.283964 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache\") pod \"3ace2724-f45d-459b-8ce2-2ee7d9542035\" (UID: \"3ace2724-f45d-459b-8ce2-2ee7d9542035\") " Jul 02 20:51:22.284156 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.284074 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:22.284265 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.284186 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home" (OuterVolumeSpecName: "home") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:22.284351 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.284320 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:22.284418 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.284389 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-tmp-dir\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:22.284418 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.284409 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-home\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:22.286532 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.286502 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm" (OuterVolumeSpecName: "kube-api-access-b7rfm") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "kube-api-access-b7rfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:51:22.286731 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.286708 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm" (OuterVolumeSpecName: "dshm") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:22.286793 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.286708 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ace2724-f45d-459b-8ce2-2ee7d9542035" (UID: "3ace2724-f45d-459b-8ce2-2ee7d9542035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:51:22.385692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.385658 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-dshm\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:22.385692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.385686 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ace2724-f45d-459b-8ce2-2ee7d9542035-model-cache\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:22.385692 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.385697 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7rfm\" (UniqueName: \"kubernetes.io/projected/3ace2724-f45d-459b-8ce2-2ee7d9542035-kube-api-access-b7rfm\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:22.385972 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:22.385706 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ace2724-f45d-459b-8ce2-2ee7d9542035-tls-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:23.160022 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:23.159987 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" Jul 02 20:51:23.160502 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:23.160016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv" event={"ID":"3ace2724-f45d-459b-8ce2-2ee7d9542035","Type":"ContainerDied","Data":"c0b2a9b13e9209bc5d498ff845e83784897e5caa9c9e75a75d2466c2fd61b57e"} Jul 02 20:51:23.160502 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:23.160068 2579 scope.go:117] "RemoveContainer" containerID="2b227a44d0df983b9958e41bede91a60db50a5a6480093eff89be6538f364e7c" Jul 02 20:51:23.177513 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:23.177476 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:51:23.180873 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:23.180831 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-237f0d/test-model-237f0d-kserve-5c5645d65c-mbfzv"] Jul 02 20:51:24.339623 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:24.339592 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ace2724-f45d-459b-8ce2-2ee7d9542035" path="/var/lib/kubelet/pods/3ace2724-f45d-459b-8ce2-2ee7d9542035/volumes" Jul 02 20:51:50.803878 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:50.803826 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:51:50.804344 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:50.804068 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" podUID="c2166837-940f-4366-8f6d-fa3696481d01" containerName="maas-api" containerID="cri-o://8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168" gracePeriod=30 Jul 02 20:51:51.252099 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.252067 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:51:51.291170 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.291137 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2166837-940f-4366-8f6d-fa3696481d01" containerID="8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168" exitCode=0 Jul 02 20:51:51.291356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.291243 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" Jul 02 20:51:51.291356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.291279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" event={"ID":"c2166837-940f-4366-8f6d-fa3696481d01","Type":"ContainerDied","Data":"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168"} Jul 02 20:51:51.291356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.291335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn" event={"ID":"c2166837-940f-4366-8f6d-fa3696481d01","Type":"ContainerDied","Data":"12a21263e152938fc5084ad132604090b61500cbc3f41ecab07de7c14488d4b6"} Jul 02 20:51:51.291531 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.291359 2579 scope.go:117] "RemoveContainer" containerID="8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168" Jul 02 20:51:51.301643 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.301620 2579 scope.go:117] "RemoveContainer" containerID="8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168" Jul 02 20:51:51.301906 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:51:51.301883 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168\": container with ID starting with 8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168 not found: ID does not exist" containerID="8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168" Jul 02 20:51:51.301998 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.301912 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168"} err="failed to get container status \"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168\": rpc error: code = NotFound desc = could not find container \"8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168\": container with ID starting with 8e81699205e4f3b6ba83841755363bf4858c531a61dfd55affcd2e30caf7b168 not found: ID does not exist" Jul 02 20:51:51.346084 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.345991 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca\") pod \"c2166837-940f-4366-8f6d-fa3696481d01\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " Jul 02 20:51:51.346084 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.346039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs7np\" (UniqueName: \"kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np\") pod \"c2166837-940f-4366-8f6d-fa3696481d01\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " Jul 02 20:51:51.346084 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.346079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") pod \"c2166837-940f-4366-8f6d-fa3696481d01\" (UID: \"c2166837-940f-4366-8f6d-fa3696481d01\") " Jul 02 20:51:51.346506 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.346477 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2166837-940f-4366-8f6d-fa3696481d01" (UID: "c2166837-940f-4366-8f6d-fa3696481d01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:51:51.348655 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.348628 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np" (OuterVolumeSpecName: "kube-api-access-xs7np") pod "c2166837-940f-4366-8f6d-fa3696481d01" (UID: "c2166837-940f-4366-8f6d-fa3696481d01"). InnerVolumeSpecName "kube-api-access-xs7np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:51:51.348757 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.348744 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2166837-940f-4366-8f6d-fa3696481d01" (UID: "c2166837-940f-4366-8f6d-fa3696481d01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:51:51.448023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.447629 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2166837-940f-4366-8f6d-fa3696481d01-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:51.448023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.447663 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7np\" (UniqueName: \"kubernetes.io/projected/c2166837-940f-4366-8f6d-fa3696481d01-kube-api-access-xs7np\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:51.448023 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.447680 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2166837-940f-4366-8f6d-fa3696481d01-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:51:51.612687 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.612650 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:51:51.616257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:51.616231 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-237f0d-5b5c8c8dfc-wdjsn"] Jul 02 20:51:52.345929 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:52.345890 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2166837-940f-4366-8f6d-fa3696481d01" path="/var/lib/kubelet/pods/c2166837-940f-4366-8f6d-fa3696481d01/volumes" Jul 02 20:51:53.998162 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:51:53.998120 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:52:00.417695 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.417660 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418043 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2166837-940f-4366-8f6d-fa3696481d01" containerName="maas-api" Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418054 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2166837-940f-4366-8f6d-fa3696481d01" containerName="maas-api" Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418064 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" containerName="maas-api" Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418069 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" containerName="maas-api" Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418079 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ace2724-f45d-459b-8ce2-2ee7d9542035" containerName="main" Jul 02 20:52:00.418103 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418084 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ace2724-f45d-459b-8ce2-2ee7d9542035" containerName="main" Jul 02 20:52:00.418321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418144 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2166837-940f-4366-8f6d-fa3696481d01" containerName="maas-api" Jul 02 20:52:00.418321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418153 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ace2724-f45d-459b-8ce2-2ee7d9542035" containerName="main" Jul 02 20:52:00.418321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.418160 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a4be9e-bfa5-4145-8326-c17f33ca70d7" containerName="maas-api" Jul 02 20:52:00.421206 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.421190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.423939 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.423915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-3f65e9-serving-cert\"" Jul 02 20:52:00.424070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.423920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:52:00.431243 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.431219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:00.530438 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.530402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.530620 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.530559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.530620 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.530595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhs2x\" (UniqueName: \"kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.631946 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.631910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.632153 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.631952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhs2x\" (UniqueName: \"kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.632153 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.632009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.632153 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:00.632116 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-3f65e9-serving-cert: secret "maas-api-e2e-api-a-3f65e9-serving-cert" not found Jul 02 20:52:00.632326 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:00.632194 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls podName:44fc1535-b85f-4eee-8b84-1139d858728e nodeName:}" failed. No retries permitted until 2026-07-02 20:52:01.132172608 +0000 UTC m=+2229.337559221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls") pod "maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" (UID: "44fc1535-b85f-4eee-8b84-1139d858728e") : secret "maas-api-e2e-api-a-3f65e9-serving-cert" not found Jul 02 20:52:00.632699 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.632680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:00.641420 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:00.641396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhs2x\" (UniqueName: \"kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:01.138016 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:01.137987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:01.138188 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:01.138133 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-3f65e9-serving-cert: secret "maas-api-e2e-api-a-3f65e9-serving-cert" not found Jul 02 20:52:01.138285 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:01.138231 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls podName:44fc1535-b85f-4eee-8b84-1139d858728e nodeName:}" failed. No retries permitted until 2026-07-02 20:52:02.138215006 +0000 UTC m=+2230.343601617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls") pod "maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" (UID: "44fc1535-b85f-4eee-8b84-1139d858728e") : secret "maas-api-e2e-api-a-3f65e9-serving-cert" not found Jul 02 20:52:02.148269 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:02.148226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:02.151054 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:02.151014 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:02.232034 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:02.231991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:02.392802 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:02.392596 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:02.395165 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:52:02.395134 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fc1535_b85f_4eee_8b84_1139d858728e.slice/crio-6028dfea8c5bf309da2081df725659f3565916dfefd205f33e02105e1a31e1d6 WatchSource:0}: Error finding container 6028dfea8c5bf309da2081df725659f3565916dfefd205f33e02105e1a31e1d6: Status 404 returned error can't find the container with id 6028dfea8c5bf309da2081df725659f3565916dfefd205f33e02105e1a31e1d6 Jul 02 20:52:03.351806 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:03.351770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" event={"ID":"44fc1535-b85f-4eee-8b84-1139d858728e","Type":"ContainerStarted","Data":"6028dfea8c5bf309da2081df725659f3565916dfefd205f33e02105e1a31e1d6"} Jul 02 20:52:04.357544 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:04.357511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" event={"ID":"44fc1535-b85f-4eee-8b84-1139d858728e","Type":"ContainerStarted","Data":"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac"} Jul 02 20:52:04.357954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:04.357653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:04.377258 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:04.377205 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" podStartSLOduration=3.328097564 podStartE2EDuration="4.377187703s" totalImagesPullingTime="1.049090139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:52:04.375487035 +0000 UTC m=+2232.580873667" watchObservedRunningTime="2026-07-02 20:52:04.377187703 +0000 UTC m=+2232.582574334" Jul 02 20:52:08.597388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.597348 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:08.601138 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.601111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.603667 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.603642 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-199208-serving-cert\"" Jul 02 20:52:08.606287 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.606256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75hq\" (UniqueName: \"kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.606388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.606320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.606388 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.606370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.625909 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.625882 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:08.707498 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.707467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75hq\" (UniqueName: \"kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.707681 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.707513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.707681 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.707545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.707681 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:08.707645 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-199208-serving-cert: secret "maas-api-e2e-api-b-199208-serving-cert" not found Jul 02 20:52:08.707806 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:08.707697 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls podName:a5b82107-bda1-494c-bd30-a50e2afd681d nodeName:}" failed. No retries permitted until 2026-07-02 20:52:09.207680617 +0000 UTC m=+2237.413067223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls") pod "maas-api-e2e-api-b-199208-58cb7f999-n8cj5" (UID: "a5b82107-bda1-494c-bd30-a50e2afd681d") : secret "maas-api-e2e-api-b-199208-serving-cert" not found Jul 02 20:52:08.708158 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.708141 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:08.717689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:08.717666 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75hq\" (UniqueName: \"kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:09.212299 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:09.212242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:09.214800 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:09.214781 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") pod \"maas-api-e2e-api-b-199208-58cb7f999-n8cj5\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:09.405954 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:09.405921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:52:09.512234 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:09.512146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:09.872125 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:09.872091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:09.873788 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:52:09.873745 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b82107_bda1_494c_bd30_a50e2afd681d.slice/crio-109e28f6082aa954e0eb2134e4c0fea957e004530cdb9ba984ccd4df27f6fe01 WatchSource:0}: Error finding container 109e28f6082aa954e0eb2134e4c0fea957e004530cdb9ba984ccd4df27f6fe01: Status 404 returned error can't find the container with id 109e28f6082aa954e0eb2134e4c0fea957e004530cdb9ba984ccd4df27f6fe01 Jul 02 20:52:10.368458 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:10.368432 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:10.386240 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:10.386211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" event={"ID":"a5b82107-bda1-494c-bd30-a50e2afd681d","Type":"ContainerStarted","Data":"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4"} Jul 02 20:52:10.386362 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:10.386256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" event={"ID":"a5b82107-bda1-494c-bd30-a50e2afd681d","Type":"ContainerStarted","Data":"109e28f6082aa954e0eb2134e4c0fea957e004530cdb9ba984ccd4df27f6fe01"} Jul 02 20:52:10.386362 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:10.386289 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:10.409803 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:10.409744 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" podStartSLOduration=1.978484904 podStartE2EDuration="2.409728725s" totalImagesPullingTime="431.243821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:52:10.408446259 +0000 UTC m=+2238.613832889" watchObservedRunningTime="2026-07-02 20:52:10.409728725 +0000 UTC m=+2238.615115403" Jul 02 20:52:16.396373 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:16.396344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:21.819359 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:21.819330 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:21.819865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:21.819519 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" podUID="44fc1535-b85f-4eee-8b84-1139d858728e" containerName="maas-api" containerID="cri-o://63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac" gracePeriod=30 Jul 02 20:52:22.073040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.072964 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:22.126708 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.126675 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhs2x\" (UniqueName: \"kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x\") pod \"44fc1535-b85f-4eee-8b84-1139d858728e\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " Jul 02 20:52:22.126942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.126749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca\") pod \"44fc1535-b85f-4eee-8b84-1139d858728e\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " Jul 02 20:52:22.126942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.126785 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") pod \"44fc1535-b85f-4eee-8b84-1139d858728e\" (UID: \"44fc1535-b85f-4eee-8b84-1139d858728e\") " Jul 02 20:52:22.127209 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.127178 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44fc1535-b85f-4eee-8b84-1139d858728e" (UID: "44fc1535-b85f-4eee-8b84-1139d858728e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:52:22.129199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.129163 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44fc1535-b85f-4eee-8b84-1139d858728e" (UID: "44fc1535-b85f-4eee-8b84-1139d858728e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:52:22.129320 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.129202 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x" (OuterVolumeSpecName: "kube-api-access-qhs2x") pod "44fc1535-b85f-4eee-8b84-1139d858728e" (UID: "44fc1535-b85f-4eee-8b84-1139d858728e"). InnerVolumeSpecName "kube-api-access-qhs2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:52:22.227728 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.227689 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44fc1535-b85f-4eee-8b84-1139d858728e-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:22.227728 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.227723 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44fc1535-b85f-4eee-8b84-1139d858728e-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:22.227728 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.227733 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhs2x\" (UniqueName: \"kubernetes.io/projected/44fc1535-b85f-4eee-8b84-1139d858728e-kube-api-access-qhs2x\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:22.443675 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.443637 2579 generic.go:356] "Generic (PLEG): container finished" podID="44fc1535-b85f-4eee-8b84-1139d858728e" containerID="63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac" exitCode=0 Jul 02 20:52:22.443897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.443697 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" Jul 02 20:52:22.443897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.443705 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" event={"ID":"44fc1535-b85f-4eee-8b84-1139d858728e","Type":"ContainerDied","Data":"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac"} Jul 02 20:52:22.443897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.443755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l" event={"ID":"44fc1535-b85f-4eee-8b84-1139d858728e","Type":"ContainerDied","Data":"6028dfea8c5bf309da2081df725659f3565916dfefd205f33e02105e1a31e1d6"} Jul 02 20:52:22.443897 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.443785 2579 scope.go:117] "RemoveContainer" containerID="63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac" Jul 02 20:52:22.455433 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.455399 2579 scope.go:117] "RemoveContainer" containerID="63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac" Jul 02 20:52:22.455683 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:22.455666 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac\": container with ID starting with 63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac not found: ID does not exist" containerID="63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac" Jul 02 20:52:22.455748 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.455690 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac"} err="failed to get container status \"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac\": rpc error: code = NotFound desc = could not find container \"63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac\": container with ID starting with 63a528bf3b13932c65681063932ebcbd08759b4cbb0f494b05d8b435e3f53fac not found: ID does not exist" Jul 02 20:52:22.460524 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.460501 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:22.464394 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.464373 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3f65e9-98499dc5-5vn5l"] Jul 02 20:52:22.804554 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:22.804520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:52:24.340665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:24.340628 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44fc1535-b85f-4eee-8b84-1139d858728e" path="/var/lib/kubelet/pods/44fc1535-b85f-4eee-8b84-1139d858728e/volumes" Jul 02 20:52:28.595415 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.594818 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:28.595963 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.595672 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" podUID="a5b82107-bda1-494c-bd30-a50e2afd681d" containerName="maas-api" containerID="cri-o://a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4" gracePeriod=30 Jul 02 20:52:28.854701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.854620 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:28.988162 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.988130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca\") pod \"a5b82107-bda1-494c-bd30-a50e2afd681d\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " Jul 02 20:52:28.988356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.988174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g75hq\" (UniqueName: \"kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq\") pod \"a5b82107-bda1-494c-bd30-a50e2afd681d\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " Jul 02 20:52:28.988356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.988305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") pod \"a5b82107-bda1-494c-bd30-a50e2afd681d\" (UID: \"a5b82107-bda1-494c-bd30-a50e2afd681d\") " Jul 02 20:52:28.988557 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.988534 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5b82107-bda1-494c-bd30-a50e2afd681d" (UID: "a5b82107-bda1-494c-bd30-a50e2afd681d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:52:28.990295 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.990266 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5b82107-bda1-494c-bd30-a50e2afd681d" (UID: "a5b82107-bda1-494c-bd30-a50e2afd681d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:52:28.990448 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:28.990373 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq" (OuterVolumeSpecName: "kube-api-access-g75hq") pod "a5b82107-bda1-494c-bd30-a50e2afd681d" (UID: "a5b82107-bda1-494c-bd30-a50e2afd681d"). InnerVolumeSpecName "kube-api-access-g75hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:52:29.089070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.089035 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5b82107-bda1-494c-bd30-a50e2afd681d-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:29.089070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.089064 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5b82107-bda1-494c-bd30-a50e2afd681d-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:29.089070 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.089075 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75hq\" (UniqueName: \"kubernetes.io/projected/a5b82107-bda1-494c-bd30-a50e2afd681d-kube-api-access-g75hq\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:52:29.481068 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.481032 2579 generic.go:356] "Generic (PLEG): container finished" podID="a5b82107-bda1-494c-bd30-a50e2afd681d" containerID="a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4" exitCode=0 Jul 02 20:52:29.481237 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.481103 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" Jul 02 20:52:29.481237 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.481119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" event={"ID":"a5b82107-bda1-494c-bd30-a50e2afd681d","Type":"ContainerDied","Data":"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4"} Jul 02 20:52:29.481237 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.481176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5" event={"ID":"a5b82107-bda1-494c-bd30-a50e2afd681d","Type":"ContainerDied","Data":"109e28f6082aa954e0eb2134e4c0fea957e004530cdb9ba984ccd4df27f6fe01"} Jul 02 20:52:29.481237 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.481199 2579 scope.go:117] "RemoveContainer" containerID="a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4" Jul 02 20:52:29.490442 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.490425 2579 scope.go:117] "RemoveContainer" containerID="a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4" Jul 02 20:52:29.490707 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:29.490688 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4\": container with ID starting with a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4 not found: ID does not exist" containerID="a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4" Jul 02 20:52:29.490759 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.490717 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4"} err="failed to get container status \"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4\": rpc error: code = NotFound desc = could not find container \"a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4\": container with ID starting with a18f120898fc189967ccf02a1261e61ea9d9e1314f5b8c79e74147fc935a59e4 not found: ID does not exist" Jul 02 20:52:29.502498 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.502465 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:29.506185 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:29.506163 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-199208-58cb7f999-n8cj5"] Jul 02 20:52:30.339489 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:30.339453 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b82107-bda1-494c-bd30-a50e2afd681d" path="/var/lib/kubelet/pods/a5b82107-bda1-494c-bd30-a50e2afd681d/volumes" Jul 02 20:52:33.005691 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:33.005656 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:52:35.556909 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.556876 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:52:35.557444 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557424 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b82107-bda1-494c-bd30-a50e2afd681d" containerName="maas-api" Jul 02 20:52:35.557531 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557447 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b82107-bda1-494c-bd30-a50e2afd681d" containerName="maas-api" Jul 02 20:52:35.557531 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557472 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44fc1535-b85f-4eee-8b84-1139d858728e" containerName="maas-api" Jul 02 20:52:35.557531 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557481 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44fc1535-b85f-4eee-8b84-1139d858728e" containerName="maas-api" Jul 02 20:52:35.557678 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557593 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44fc1535-b85f-4eee-8b84-1139d858728e" containerName="maas-api" Jul 02 20:52:35.557678 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.557611 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b82107-bda1-494c-bd30-a50e2afd681d" containerName="maas-api" Jul 02 20:52:35.562345 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.562319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.565107 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.565087 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-bf7336-openshift-default-dockercfg-l5m6h\"" Jul 02 20:52:35.572878 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.572826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:52:35.645662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.645921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.646110 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.645992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.646110 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.646014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgkbh\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747239 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747239 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgkbh\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747745 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.747935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.747806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.748106 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.748085 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.748320 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.748301 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.750058 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.750030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.750619 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.750596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.756672 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.756622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.757002 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.756969 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgkbh\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh\") pod \"e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:35.873929 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:35.873899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:36.016777 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.016751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:52:36.019001 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:52:36.018968 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0ab3e0_381b_4c8a_b630_c8fe1d08e02f.slice/crio-75526191b980557d45f691f6bed70e75e237fec795d8f8b83f506677c7364e81 WatchSource:0}: Error finding container 75526191b980557d45f691f6bed70e75e237fec795d8f8b83f506677c7364e81: Status 404 returned error can't find the container with id 75526191b980557d45f691f6bed70e75e237fec795d8f8b83f506677c7364e81 Jul 02 20:52:36.021140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.021109 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:36.021257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.021173 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:36.021257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.021199 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:36.524195 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.524163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerStarted","Data":"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4"} Jul 02 20:52:36.524368 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.524216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerStarted","Data":"75526191b980557d45f691f6bed70e75e237fec795d8f8b83f506677c7364e81"} Jul 02 20:52:36.548063 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.547995 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podStartSLOduration=1.547972212 podStartE2EDuration="1.547972212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:52:36.546586667 +0000 UTC m=+2264.751973317" watchObservedRunningTime="2026-07-02 20:52:36.547972212 +0000 UTC m=+2264.753358842" Jul 02 20:52:36.874283 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:36.874250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:37.879637 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:37.879608 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:38.534200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.534171 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:38.535334 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.535308 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:38.604408 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.604378 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:52:38.607997 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.607977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.610503 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.610478 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-bf7336-serving-cert\"" Jul 02 20:52:38.610676 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.610658 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:52:38.617647 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.617615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:52:38.676984 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.676941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.677199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.677080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.677199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.677117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8mf\" (UniqueName: \"kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.778230 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.778185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.778422 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.778252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.778422 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.778271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8mf\" (UniqueName: \"kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.778422 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:38.778336 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-bf7336-serving-cert: secret "maas-api-e2e-shared-a-bf7336-serving-cert" not found Jul 02 20:52:38.778422 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:38.778418 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls podName:34640869-1f47-43ef-8bd3-bbc23804e6a9 nodeName:}" failed. No retries permitted until 2026-07-02 20:52:39.278402323 +0000 UTC m=+2267.483788930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls") pod "maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" (UID: "34640869-1f47-43ef-8bd3-bbc23804e6a9") : secret "maas-api-e2e-shared-a-bf7336-serving-cert" not found Jul 02 20:52:38.778947 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.778930 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:38.789702 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:38.789623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8mf\" (UniqueName: \"kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:39.282463 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:39.282428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:39.285176 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:39.285143 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") pod \"maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:39.521810 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:39.521766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:39.667437 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:39.667409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:52:39.669943 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:52:39.669904 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34640869_1f47_43ef_8bd3_bbc23804e6a9.slice/crio-32b409d4025e0b8664b14c357f8d32492b4bdcedd19d809b7f3f6fbbf7cf24f0 WatchSource:0}: Error finding container 32b409d4025e0b8664b14c357f8d32492b4bdcedd19d809b7f3f6fbbf7cf24f0: Status 404 returned error can't find the container with id 32b409d4025e0b8664b14c357f8d32492b4bdcedd19d809b7f3f6fbbf7cf24f0 Jul 02 20:52:40.549886 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:40.549814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" event={"ID":"34640869-1f47-43ef-8bd3-bbc23804e6a9","Type":"ContainerStarted","Data":"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849"} Jul 02 20:52:40.550382 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:40.549903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" event={"ID":"34640869-1f47-43ef-8bd3-bbc23804e6a9","Type":"ContainerStarted","Data":"32b409d4025e0b8664b14c357f8d32492b4bdcedd19d809b7f3f6fbbf7cf24f0"} Jul 02 20:52:40.550382 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:40.550131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:40.567099 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:40.567044 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" podStartSLOduration=2.286204996 podStartE2EDuration="2.567027077s" totalImagesPullingTime="280.822081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:52:40.565480322 +0000 UTC m=+2268.770866972" watchObservedRunningTime="2026-07-02 20:52:40.567027077 +0000 UTC m=+2268.772413685" Jul 02 20:52:44.503266 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:44.503231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:52:46.574868 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.574279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:52:46.794308 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.794278 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:52:46.798229 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.798205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:46.800591 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.800565 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-9994a6-serving-cert\"" Jul 02 20:52:46.809431 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.809406 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:52:46.973089 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.973032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:46.973089 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.973078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9kz\" (UniqueName: \"kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:46.973370 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:46.973174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.074353 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.074306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.074557 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.074358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9kz\" (UniqueName: \"kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.074557 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.074422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.074557 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:47.074470 2579 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-9994a6-serving-cert: secret "maas-api-e2e-shared-b-9994a6-serving-cert" not found Jul 02 20:52:47.074557 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:52:47.074533 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls podName:1e4ab381-a9fc-45e0-8624-c341bc081b67 nodeName:}" failed. No retries permitted until 2026-07-02 20:52:47.574515967 +0000 UTC m=+2275.779902574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls") pod "maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" (UID: "1e4ab381-a9fc-45e0-8624-c341bc081b67") : secret "maas-api-e2e-shared-b-9994a6-serving-cert" not found Jul 02 20:52:47.075200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.075176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.092584 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.092555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9kz\" (UniqueName: \"kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.580053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.580017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.582857 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.582810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") pod \"maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.709733 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.709692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:47.895956 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:47.895928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:52:47.898744 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:52:47.898716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4ab381_a9fc_45e0_8624_c341bc081b67.slice/crio-90372209d2ed7b97d674f88b0ee88f102e80053d99a8a4612cac1f82623f0747 WatchSource:0}: Error finding container 90372209d2ed7b97d674f88b0ee88f102e80053d99a8a4612cac1f82623f0747: Status 404 returned error can't find the container with id 90372209d2ed7b97d674f88b0ee88f102e80053d99a8a4612cac1f82623f0747 Jul 02 20:52:48.592670 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:48.592636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" event={"ID":"1e4ab381-a9fc-45e0-8624-c341bc081b67","Type":"ContainerStarted","Data":"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab"} Jul 02 20:52:48.593094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:48.592689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" event={"ID":"1e4ab381-a9fc-45e0-8624-c341bc081b67","Type":"ContainerStarted","Data":"90372209d2ed7b97d674f88b0ee88f102e80053d99a8a4612cac1f82623f0747"} Jul 02 20:52:48.593094 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:48.592770 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:48.614639 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:48.614578 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" podStartSLOduration=2.139776408 podStartE2EDuration="2.614564389s" totalImagesPullingTime="474.787981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:52:48.612081935 +0000 UTC m=+2276.817468564" watchObservedRunningTime="2026-07-02 20:52:48.614564389 +0000 UTC m=+2276.819951017" Jul 02 20:52:53.535463 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.535422 2579 patch_prober.go:28] interesting pod/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 02 20:52:53.535941 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.535479 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 02 20:52:53.620513 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.620480 2579 generic.go:356] "Generic (PLEG): container finished" podID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerID="bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4" exitCode=0 Jul 02 20:52:53.620677 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.620556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerDied","Data":"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4"} Jul 02 20:52:53.620980 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.620964 2579 scope.go:117] "RemoveContainer" containerID="bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4" Jul 02 20:52:53.621940 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.621912 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:53.622040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.621977 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:53.622040 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:53.622021 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 20:52:54.604762 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:54.604729 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:52:54.628744 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:54.628710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerStarted","Data":"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420"} Jul 02 20:52:54.874822 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:54.874781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:54.875076 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:54.875047 2579 patch_prober.go:28] interesting pod/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 02 20:52:54.875175 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:54.875108 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 02 20:52:55.874961 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:55.874924 2579 patch_prober.go:28] interesting pod/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 02 20:52:55.875346 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:55.875017 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 02 20:52:56.875047 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:56.875006 2579 patch_prober.go:28] interesting pod/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 02 20:52:56.875430 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:56.875070 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 02 20:52:57.874528 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:57.874485 2579 patch_prober.go:28] interesting pod/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 02 20:52:57.874717 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:57.874559 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 02 20:52:58.879043 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:58.879010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:58.879452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:58.879369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:52:58.880064 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:52:58.880049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:53:00.198630 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:00.198592 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:53:15.398685 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:15.398648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:53:30.606917 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.606886 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:53:30.610627 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.610607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.614352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.614328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"openshift-service-ca.crt\"" Jul 02 20:53:30.614352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.614350 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"kube-root-ca.crt\"" Jul 02 20:53:30.614546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.614385 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"auth-test-model-9994a6-kserve-self-signed-certs\"" Jul 02 20:53:30.614546 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.614333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"default-dockercfg-wr24g\"" Jul 02 20:53:30.622021 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.622000 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:53:30.690080 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.690249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.690249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.690249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.690249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.690398 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.690359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bsd\" (UniqueName: \"kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791273 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bsd\" (UniqueName: \"kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791480 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791779 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791871 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.791964 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.791943 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.793978 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.793960 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.794085 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.794066 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.799747 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.799717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bsd\" (UniqueName: \"kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd\") pod \"auth-test-model-9994a6-kserve-5694c9466c-mstjv\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:30.920689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:30.920647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:31.059658 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:31.059630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:53:31.063371 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:53:31.063334 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64da20f0_d6c4_4825_adf7_03e09584c852.slice/crio-19bb8ff7f971c2b8e776fe729deb62607c43e321ff8c50e065123eceadda165f WatchSource:0}: Error finding container 19bb8ff7f971c2b8e776fe729deb62607c43e321ff8c50e065123eceadda165f: Status 404 returned error can't find the container with id 19bb8ff7f971c2b8e776fe729deb62607c43e321ff8c50e065123eceadda165f Jul 02 20:53:31.822661 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:31.822625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" event={"ID":"64da20f0-d6c4-4825-adf7-03e09584c852","Type":"ContainerStarted","Data":"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775"} Jul 02 20:53:31.823069 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:31.822672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" event={"ID":"64da20f0-d6c4-4825-adf7-03e09584c852","Type":"ContainerStarted","Data":"19bb8ff7f971c2b8e776fe729deb62607c43e321ff8c50e065123eceadda165f"} Jul 02 20:53:31.840367 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:31.840316 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" podStartSLOduration=1.840302408 podStartE2EDuration="1.840302408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:53:31.839326668 +0000 UTC m=+2320.044713291" watchObservedRunningTime="2026-07-02 20:53:31.840302408 +0000 UTC m=+2320.045689037" Jul 02 20:53:34.705905 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:34.705867 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:53:40.920782 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:40.920741 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:40.921262 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:40.920827 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:40.933681 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:40.933650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:41.884074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:41.884044 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:53:48.708672 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:53:48.708580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:54:07.306730 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:07.306698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:54:24.703784 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:24.703745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:54:41.807768 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:41.807737 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:54:56.276506 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:56.276472 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:54:56.283718 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:56.283698 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mrg6r_e2f85b4d-8523-46da-b6db-d0fba83ed62d/ovn-acl-logging/0.log" Jul 02 20:54:59.900321 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:54:59.900278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:55:14.202356 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:14.202318 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:55:28.606670 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:28.606639 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:55:32.896804 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:32.896773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:55:43.627788 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.627741 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" podUID="64da20f0-d6c4-4825-adf7-03e09584c852" containerName="main" containerID="cri-o://d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775" gracePeriod=45 Jul 02 20:55:43.914482 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.914450 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:55:43.988464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988431 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78bsd\" (UniqueName: \"kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988464 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988473 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988491 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988516 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988602 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm\") pod \"64da20f0-d6c4-4825-adf7-03e09584c852\" (UID: \"64da20f0-d6c4-4825-adf7-03e09584c852\") " Jul 02 20:55:43.988990 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988762 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.988990 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988892 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home" (OuterVolumeSpecName: "home") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.988990 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.988973 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-tmp-dir\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.989162 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.989023 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache" (OuterVolumeSpecName: "model-cache") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.990862 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.990793 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm" (OuterVolumeSpecName: "dshm") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.990992 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.990918 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd" (OuterVolumeSpecName: "kube-api-access-78bsd") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "kube-api-access-78bsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:55:43.990992 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:43.990918 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64da20f0-d6c4-4825-adf7-03e09584c852" (UID: "64da20f0-d6c4-4825-adf7-03e09584c852"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:55:44.090352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.090317 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64da20f0-d6c4-4825-adf7-03e09584c852-tls-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:44.090352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.090344 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-home\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:44.090352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.090354 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-model-cache\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:44.090352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.090362 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64da20f0-d6c4-4825-adf7-03e09584c852-dshm\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:44.090604 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.090370 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bsd\" (UniqueName: \"kubernetes.io/projected/64da20f0-d6c4-4825-adf7-03e09584c852-kube-api-access-78bsd\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:55:44.495072 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.495032 2579 generic.go:356] "Generic (PLEG): container finished" podID="64da20f0-d6c4-4825-adf7-03e09584c852" containerID="d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775" exitCode=0 Jul 02 20:55:44.495249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.495090 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" Jul 02 20:55:44.495249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.495128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" event={"ID":"64da20f0-d6c4-4825-adf7-03e09584c852","Type":"ContainerDied","Data":"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775"} Jul 02 20:55:44.495249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.495176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv" event={"ID":"64da20f0-d6c4-4825-adf7-03e09584c852","Type":"ContainerDied","Data":"19bb8ff7f971c2b8e776fe729deb62607c43e321ff8c50e065123eceadda165f"} Jul 02 20:55:44.495249 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.495193 2579 scope.go:117] "RemoveContainer" containerID="d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775" Jul 02 20:55:44.504045 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.504025 2579 scope.go:117] "RemoveContainer" containerID="d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775" Jul 02 20:55:44.504323 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:55:44.504285 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775\": container with ID starting with d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775 not found: ID does not exist" containerID="d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775" Jul 02 20:55:44.504370 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.504321 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775"} err="failed to get container status \"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775\": rpc error: code = NotFound desc = could not find container \"d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775\": container with ID starting with d4775666624681a533740a4cc31966cd05964d07500d2477d22cb3873e5ee775 not found: ID does not exist" Jul 02 20:55:44.513087 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.513062 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:55:44.517532 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:44.517512 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/auth-test-model-9994a6-kserve-5694c9466c-mstjv"] Jul 02 20:55:46.341055 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:46.341024 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64da20f0-d6c4-4825-adf7-03e09584c852" path="/var/lib/kubelet/pods/64da20f0-d6c4-4825-adf7-03e09584c852/volumes" Jul 02 20:55:49.798440 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:55:49.798407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:56:06.902963 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:56:06.902928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:56:27.755322 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:56:27.755286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:56:47.603934 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:56:47.603830 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:57:05.505657 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:05.505607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:57:26.096950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:26.096912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:57:43.615014 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.614984 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:57:43.615421 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.615361 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64da20f0-d6c4-4825-adf7-03e09584c852" containerName="main" Jul 02 20:57:43.615421 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.615371 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64da20f0-d6c4-4825-adf7-03e09584c852" containerName="main" Jul 02 20:57:43.615501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.615445 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64da20f0-d6c4-4825-adf7-03e09584c852" containerName="main" Jul 02 20:57:43.618290 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.618273 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.621364 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.621342 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"rate-test-model-9994a6-kserve-self-signed-certs\"" Jul 02 20:57:43.622343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.622325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"default-dockercfg-wr24g\"" Jul 02 20:57:43.622449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.622382 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"openshift-service-ca.crt\"" Jul 02 20:57:43.622449 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.622430 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"kube-root-ca.crt\"" Jul 02 20:57:43.635000 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.634975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:57:43.789935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.789898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.789935 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.789934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.790139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.789993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.790139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.790038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.790139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.790087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.790139 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.790106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f276l\" (UniqueName: \"kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891200 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f276l\" (UniqueName: \"kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891494 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891494 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891494 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.891660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.891619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.893978 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.893949 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.894110 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.894082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.905501 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.905468 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f276l\" (UniqueName: \"kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l\") pod \"rate-test-model-9994a6-kserve-54cd95b659-2nf9x\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:43.928005 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:43.927979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:44.078811 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:44.078770 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:57:44.082766 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:57:44.082737 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11eced68_100c_4003_96a4_cf21afb9161b.slice/crio-93e93cc6eda07fd64449f11afed1173bff6ba183ad5e2354f503429c501d3ac7 WatchSource:0}: Error finding container 93e93cc6eda07fd64449f11afed1173bff6ba183ad5e2354f503429c501d3ac7: Status 404 returned error can't find the container with id 93e93cc6eda07fd64449f11afed1173bff6ba183ad5e2354f503429c501d3ac7 Jul 02 20:57:44.092904 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:44.092876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" event={"ID":"11eced68-100c-4003-96a4-cf21afb9161b","Type":"ContainerStarted","Data":"93e93cc6eda07fd64449f11afed1173bff6ba183ad5e2354f503429c501d3ac7"} Jul 02 20:57:45.100648 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:45.100610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" event={"ID":"11eced68-100c-4003-96a4-cf21afb9161b","Type":"ContainerStarted","Data":"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6"} Jul 02 20:57:45.119649 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:45.119608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" podStartSLOduration=2.119593373 podStartE2EDuration="2.119593373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:57:45.118173684 +0000 UTC m=+2573.323560315" watchObservedRunningTime="2026-07-02 20:57:45.119593373 +0000 UTC m=+2573.324980001" Jul 02 20:57:47.821263 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:47.821221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:57:53.928955 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:53.928914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:53.931310 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:53.928992 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:53.941167 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:53.941150 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:57:54.156411 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:57:54.156382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:58:03.499605 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:03.499564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:58:23.093068 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:23.093031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:58:30.713895 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:30.713828 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:58:38.698391 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:38.698345 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:58:45.735666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:45.735588 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" podUID="11eced68-100c-4003-96a4-cf21afb9161b" containerName="main" containerID="cri-o://6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6" gracePeriod=45 Jul 02 20:58:45.979221 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:45.979194 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:58:46.083521 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083434 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083521 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083521 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083554 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f276l\" (UniqueName: \"kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083627 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home\") pod \"11eced68-100c-4003-96a4-cf21afb9161b\" (UID: \"11eced68-100c-4003-96a4-cf21afb9161b\") " Jul 02 20:58:46.083819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083760 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:46.083819 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083778 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache" (OuterVolumeSpecName: "model-cache") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:46.084073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083958 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home" (OuterVolumeSpecName: "home") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:46.084073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.083986 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-tmp-dir\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.084073 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.084009 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-model-cache\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.085943 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.085919 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l" (OuterVolumeSpecName: "kube-api-access-f276l") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "kube-api-access-f276l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:46.086053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.085945 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm" (OuterVolumeSpecName: "dshm") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:46.086053 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.085943 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11eced68-100c-4003-96a4-cf21afb9161b" (UID: "11eced68-100c-4003-96a4-cf21afb9161b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:58:46.184942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.184906 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-home\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.184942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.184935 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11eced68-100c-4003-96a4-cf21afb9161b-tls-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.184942 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.184947 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f276l\" (UniqueName: \"kubernetes.io/projected/11eced68-100c-4003-96a4-cf21afb9161b-kube-api-access-f276l\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.185182 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.184958 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11eced68-100c-4003-96a4-cf21afb9161b-dshm\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:46.401577 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.401545 2579 generic.go:356] "Generic (PLEG): container finished" podID="11eced68-100c-4003-96a4-cf21afb9161b" containerID="6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6" exitCode=0 Jul 02 20:58:46.401751 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.401598 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" Jul 02 20:58:46.401751 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.401626 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" event={"ID":"11eced68-100c-4003-96a4-cf21afb9161b","Type":"ContainerDied","Data":"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6"} Jul 02 20:58:46.401751 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.401680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x" event={"ID":"11eced68-100c-4003-96a4-cf21afb9161b","Type":"ContainerDied","Data":"93e93cc6eda07fd64449f11afed1173bff6ba183ad5e2354f503429c501d3ac7"} Jul 02 20:58:46.401751 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.401699 2579 scope.go:117] "RemoveContainer" containerID="6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6" Jul 02 20:58:46.410450 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.410432 2579 scope.go:117] "RemoveContainer" containerID="6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6" Jul 02 20:58:46.410708 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:58:46.410687 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6\": container with ID starting with 6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6 not found: ID does not exist" containerID="6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6" Jul 02 20:58:46.410760 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.410717 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6"} err="failed to get container status \"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6\": rpc error: code = NotFound desc = could not find container \"6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6\": container with ID starting with 6640eac9762853efd29a8989c8801bd39f1a16b8bf70182e5fde8920e44046d6 not found: ID does not exist" Jul 02 20:58:46.419074 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.419050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:58:46.422310 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:46.422289 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/rate-test-model-9994a6-kserve-54cd95b659-2nf9x"] Jul 02 20:58:48.339858 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:48.339817 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11eced68-100c-4003-96a4-cf21afb9161b" path="/var/lib/kubelet/pods/11eced68-100c-4003-96a4-cf21afb9161b/volumes" Jul 02 20:58:56.620539 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:56.620504 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:58:56.622178 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:56.622146 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" podUID="34640869-1f47-43ef-8bd3-bbc23804e6a9" containerName="maas-api" containerID="cri-o://0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849" gracePeriod=30 Jul 02 20:58:57.079636 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.079610 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:58:57.186631 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.186519 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt8mf\" (UniqueName: \"kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf\") pod \"34640869-1f47-43ef-8bd3-bbc23804e6a9\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " Jul 02 20:58:57.186631 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.186623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca\") pod \"34640869-1f47-43ef-8bd3-bbc23804e6a9\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " Jul 02 20:58:57.186925 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.186699 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") pod \"34640869-1f47-43ef-8bd3-bbc23804e6a9\" (UID: \"34640869-1f47-43ef-8bd3-bbc23804e6a9\") " Jul 02 20:58:57.186988 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.186950 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34640869-1f47-43ef-8bd3-bbc23804e6a9" (UID: "34640869-1f47-43ef-8bd3-bbc23804e6a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:58:57.188792 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.188771 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34640869-1f47-43ef-8bd3-bbc23804e6a9" (UID: "34640869-1f47-43ef-8bd3-bbc23804e6a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:58:57.188904 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.188881 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf" (OuterVolumeSpecName: "kube-api-access-kt8mf") pod "34640869-1f47-43ef-8bd3-bbc23804e6a9" (UID: "34640869-1f47-43ef-8bd3-bbc23804e6a9"). InnerVolumeSpecName "kube-api-access-kt8mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:57.288261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.288210 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34640869-1f47-43ef-8bd3-bbc23804e6a9-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:57.288261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.288256 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8mf\" (UniqueName: \"kubernetes.io/projected/34640869-1f47-43ef-8bd3-bbc23804e6a9-kube-api-access-kt8mf\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:57.288261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.288267 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34640869-1f47-43ef-8bd3-bbc23804e6a9-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:58:57.457465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.457375 2579 generic.go:356] "Generic (PLEG): container finished" podID="34640869-1f47-43ef-8bd3-bbc23804e6a9" containerID="0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849" exitCode=0 Jul 02 20:58:57.457465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.457422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" event={"ID":"34640869-1f47-43ef-8bd3-bbc23804e6a9","Type":"ContainerDied","Data":"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849"} Jul 02 20:58:57.457465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.457438 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" Jul 02 20:58:57.457729 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.457469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l" event={"ID":"34640869-1f47-43ef-8bd3-bbc23804e6a9","Type":"ContainerDied","Data":"32b409d4025e0b8664b14c357f8d32492b4bdcedd19d809b7f3f6fbbf7cf24f0"} Jul 02 20:58:57.457729 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.457491 2579 scope.go:117] "RemoveContainer" containerID="0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849" Jul 02 20:58:57.469551 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.469526 2579 scope.go:117] "RemoveContainer" containerID="0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849" Jul 02 20:58:57.469896 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:58:57.469875 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849\": container with ID starting with 0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849 not found: ID does not exist" containerID="0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849" Jul 02 20:58:57.469968 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.469910 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849"} err="failed to get container status \"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849\": rpc error: code = NotFound desc = could not find container \"0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849\": container with ID starting with 0a9e877d0079fb596003dfb688ba0d1c0c26d274508126e35003018e2a23a849 not found: ID does not exist" Jul 02 20:58:57.481593 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.481566 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:58:57.486025 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:57.486001 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-bf7336-676c96444f-vlk6l"] Jul 02 20:58:58.340498 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:58.340457 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34640869-1f47-43ef-8bd3-bbc23804e6a9" path="/var/lib/kubelet/pods/34640869-1f47-43ef-8bd3-bbc23804e6a9/volumes" Jul 02 20:58:59.701195 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:58:59.701157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:59:02.551712 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:02.551678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:59:02.552238 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:02.551917 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" containerID="cri-o://168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420" gracePeriod=30 Jul 02 20:59:03.601172 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:03.601141 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:59:03.601923 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:03.601890 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerName="maas-api" containerID="cri-o://f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab" gracePeriod=30 Jul 02 20:59:04.599261 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.599222 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.49:8443/health\": dial tcp 10.134.0.49:8443: connect: connection refused" Jul 02 20:59:04.863199 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.863175 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:59:04.959964 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.959936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca\") pod \"1e4ab381-a9fc-45e0-8624-c341bc081b67\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " Jul 02 20:59:04.960132 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.960010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") pod \"1e4ab381-a9fc-45e0-8624-c341bc081b67\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " Jul 02 20:59:04.960132 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.960034 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9kz\" (UniqueName: \"kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz\") pod \"1e4ab381-a9fc-45e0-8624-c341bc081b67\" (UID: \"1e4ab381-a9fc-45e0-8624-c341bc081b67\") " Jul 02 20:59:04.960337 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.960311 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e4ab381-a9fc-45e0-8624-c341bc081b67" (UID: "1e4ab381-a9fc-45e0-8624-c341bc081b67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:59:04.962209 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.962186 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e4ab381-a9fc-45e0-8624-c341bc081b67" (UID: "1e4ab381-a9fc-45e0-8624-c341bc081b67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:59:04.962296 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:04.962255 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz" (OuterVolumeSpecName: "kube-api-access-dm9kz") pod "1e4ab381-a9fc-45e0-8624-c341bc081b67" (UID: "1e4ab381-a9fc-45e0-8624-c341bc081b67"). InnerVolumeSpecName "kube-api-access-dm9kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:59:05.061259 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.061223 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e4ab381-a9fc-45e0-8624-c341bc081b67-openshift-service-ca\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:05.061259 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.061256 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e4ab381-a9fc-45e0-8624-c341bc081b67-maas-api-tls\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:05.061454 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.061270 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9kz\" (UniqueName: \"kubernetes.io/projected/1e4ab381-a9fc-45e0-8624-c341bc081b67-kube-api-access-dm9kz\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:05.496497 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.496456 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerID="f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab" exitCode=0 Jul 02 20:59:05.496665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.496508 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" Jul 02 20:59:05.496665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.496531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" event={"ID":"1e4ab381-a9fc-45e0-8624-c341bc081b67","Type":"ContainerDied","Data":"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab"} Jul 02 20:59:05.496665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.496584 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d" event={"ID":"1e4ab381-a9fc-45e0-8624-c341bc081b67","Type":"ContainerDied","Data":"90372209d2ed7b97d674f88b0ee88f102e80053d99a8a4612cac1f82623f0747"} Jul 02 20:59:05.496665 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.496614 2579 scope.go:117] "RemoveContainer" containerID="f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab" Jul 02 20:59:05.506088 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.506070 2579 scope.go:117] "RemoveContainer" containerID="f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab" Jul 02 20:59:05.506379 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:59:05.506360 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab\": container with ID starting with f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab not found: ID does not exist" containerID="f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab" Jul 02 20:59:05.506444 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.506392 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab"} err="failed to get container status \"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab\": rpc error: code = NotFound desc = could not find container \"f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab\": container with ID starting with f232d7e0601f345e2d8fd1761a47200c143357d534d171de20c4a9db81ba89ab not found: ID does not exist" Jul 02 20:59:05.517877 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.517827 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:59:05.522358 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:05.522335 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-9994a6-559667cdb9-72x9d"] Jul 02 20:59:06.340129 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:06.340099 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" path="/var/lib/kubelet/pods/1e4ab381-a9fc-45e0-8624-c341bc081b67/volumes" Jul 02 20:59:07.806229 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.806200 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:59:07.884587 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.884764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.884764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884639 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.884764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.884764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884679 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgkbh\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.884764 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884724 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.885130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884778 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.885130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.885130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo\") pod \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\" (UID: \"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f\") " Jul 02 20:59:07.885130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.884996 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:59:07.885130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885014 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:59:07.885417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885259 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-credential-socket\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.885417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885279 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-socket\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.885417 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885367 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:59:07.885728 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885546 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:59:07.885962 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.885936 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:59:07.887052 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.887030 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:59:07.887052 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.887031 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 02 20:59:07.887168 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.887052 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh" (OuterVolumeSpecName: "kube-api-access-pgkbh") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "kube-api-access-pgkbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:59:07.887262 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.887246 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" (UID: "4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:59:07.986211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986182 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-data\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986207 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgkbh\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-kube-api-access-pgkbh\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986218 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-token\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986226 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-workload-certs\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986235 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istiod-ca-cert\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986243 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-podinfo\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:07.986452 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:07.986250 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f-istio-envoy\") on node \"ip-10-0-139-106.ec2.internal\" DevicePath \"\"" Jul 02 20:59:08.513791 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.513754 2579 generic.go:356] "Generic (PLEG): container finished" podID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerID="168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420" exitCode=0 Jul 02 20:59:08.513950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.513806 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" Jul 02 20:59:08.513950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.513819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerDied","Data":"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420"} Jul 02 20:59:08.513950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.513897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6" event={"ID":"4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f","Type":"ContainerDied","Data":"75526191b980557d45f691f6bed70e75e237fec795d8f8b83f506677c7364e81"} Jul 02 20:59:08.513950 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.513921 2579 scope.go:117] "RemoveContainer" containerID="168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420" Jul 02 20:59:08.522536 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.522519 2579 scope.go:117] "RemoveContainer" containerID="bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4" Jul 02 20:59:08.531148 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.531132 2579 scope.go:117] "RemoveContainer" containerID="168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420" Jul 02 20:59:08.531474 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:59:08.531449 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420\": container with ID starting with 168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420 not found: ID does not exist" containerID="168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420" Jul 02 20:59:08.531540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.531486 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420"} err="failed to get container status \"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420\": rpc error: code = NotFound desc = could not find container \"168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420\": container with ID starting with 168cbd0a072b23d8d75af0ee746866152076f928d19dae636341f2661009c420 not found: ID does not exist" Jul 02 20:59:08.531540 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.531512 2579 scope.go:117] "RemoveContainer" containerID="bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4" Jul 02 20:59:08.531816 ip-10-0-139-106 kubenswrapper[2579]: E0702 20:59:08.531797 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4\": container with ID starting with bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4 not found: ID does not exist" containerID="bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4" Jul 02 20:59:08.531896 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.531822 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4"} err="failed to get container status \"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4\": rpc error: code = NotFound desc = could not find container \"bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4\": container with ID starting with bc94874aab3b0813936b051410fbd27bbf414dbb1314269e5147b1ceb186c1b4 not found: ID does not exist" Jul 02 20:59:08.532100 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.532083 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:59:08.535931 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:08.535911 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-bf7336-openshift-default-68b6b754cf-lcdq6"] Jul 02 20:59:10.079392 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:10.079359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc4b4dc57-xpbmv_ca470f80-7437-4b4c-af5e-3c0137405e6a/authorino/0.log" Jul 02 20:59:10.339640 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:10.339562 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" path="/var/lib/kubelet/pods/4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f/volumes" Jul 02 20:59:12.593737 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:12.593703 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fnx8p"] Jul 02 20:59:14.656550 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:14.656515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r7b44_d4056c24-33f2-48d9-813f-5b001e4b9e35/manager/0.log" Jul 02 20:59:14.779783 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:14.779751 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-th9ph_f41800cf-ceac-4609-a972-d459ea3ea166/manager/0.log" Jul 02 20:59:15.371888 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:15.371828 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ckzps_fc4ea332-04e9-4127-9483-8196f79f50e7/server/0.log" Jul 02 20:59:15.489683 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:15.489650 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4mk8s_c35bd262-954b-4fa0-9981-dccf2efedd71/manager/0.log" Jul 02 20:59:15.853924 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:15.853825 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-ksc2n_11751f38-8ff4-4712-9f3a-36f70e815271/copy-manifests/0.log" Jul 02 20:59:15.866569 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:15.866542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-ksc2n_11751f38-8ff4-4712-9f3a-36f70e815271/manager/0.log" Jul 02 20:59:17.257707 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.257676 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc4b4dc57-xpbmv_ca470f80-7437-4b4c-af5e-3c0137405e6a/authorino/0.log" Jul 02 20:59:17.376054 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.376024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pzxrv_5bde44fc-d3a3-4236-bcc6-eea5acf608c6/manager/0.log" Jul 02 20:59:17.488343 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.488311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qskkq_da68f9e7-997a-4697-96c7-c6983f10a42a/manager/0.log" Jul 02 20:59:17.708430 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.708397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wc5gs_a962f7e1-3278-45bc-b5c4-a26b54d24f00/registry-server/0.log" Jul 02 20:59:17.828969 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.828939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ncrdf_2e95eb86-62a3-4347-957c-d068324317de/manager/0.log" Jul 02 20:59:17.946126 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:17.946104 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fnx8p_a0d16f89-a00d-4eb8-9ac7-678858640de7/limitador/0.log" Jul 02 20:59:18.064781 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:18.064701 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-65sz6_6ce73322-ad0c-49d7-b489-47ce3b7798b6/manager/0.log" Jul 02 20:59:18.749027 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:18.748999 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-fd64s_27691e00-296f-40e2-9c6e-09bd198c10f3/kube-auth-proxy/0.log" Jul 02 20:59:18.868493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:18.868462 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/1.log" Jul 02 20:59:19.901179 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:19.901139 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbljxgp_1ca713c6-1c7d-4cbe-9d97-94696973339e/main/0.log" Jul 02 20:59:24.156086 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156047 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9mtnv/must-gather-lg64z"] Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156419 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11eced68-100c-4003-96a4-cf21afb9161b" containerName="main" Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156431 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11eced68-100c-4003-96a4-cf21afb9161b" containerName="main" Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156442 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34640869-1f47-43ef-8bd3-bbc23804e6a9" containerName="maas-api" Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156448 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34640869-1f47-43ef-8bd3-bbc23804e6a9" containerName="maas-api" Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156460 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.156475 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156465 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156480 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156486 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156493 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerName="maas-api" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156499 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerName="maas-api" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156552 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11eced68-100c-4003-96a4-cf21afb9161b" containerName="main" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156559 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156565 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4ab381-a9fc-45e0-8624-c341bc081b67" containerName="maas-api" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156571 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34640869-1f47-43ef-8bd3-bbc23804e6a9" containerName="maas-api" Jul 02 20:59:24.156662 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.156577 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0ab3e0-381b-4c8a-b630-c8fe1d08e02f" containerName="istio-proxy" Jul 02 20:59:24.161341 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.161321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.164404 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.164360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9mtnv\"/\"default-dockercfg-5s2gd\"" Jul 02 20:59:24.164547 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.164509 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9mtnv\"/\"openshift-service-ca.crt\"" Jul 02 20:59:24.165444 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.165426 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9mtnv\"/\"kube-root-ca.crt\"" Jul 02 20:59:24.168393 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.168365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/must-gather-lg64z"] Jul 02 20:59:24.234067 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.234029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/724903fe-099b-42e9-999c-bb0c54522916-must-gather-output\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.234257 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.234100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrchr\" (UniqueName: \"kubernetes.io/projected/724903fe-099b-42e9-999c-bb0c54522916-kube-api-access-nrchr\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.334583 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.334539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrchr\" (UniqueName: \"kubernetes.io/projected/724903fe-099b-42e9-999c-bb0c54522916-kube-api-access-nrchr\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.334786 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.334601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/724903fe-099b-42e9-999c-bb0c54522916-must-gather-output\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.334977 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.334958 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/724903fe-099b-42e9-999c-bb0c54522916-must-gather-output\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.343600 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.343574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrchr\" (UniqueName: \"kubernetes.io/projected/724903fe-099b-42e9-999c-bb0c54522916-kube-api-access-nrchr\") pod \"must-gather-lg64z\" (UID: \"724903fe-099b-42e9-999c-bb0c54522916\") " pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.471538 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.471450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/must-gather-lg64z" Jul 02 20:59:24.605214 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.605189 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/must-gather-lg64z"] Jul 02 20:59:24.607289 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:59:24.607262 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod724903fe_099b_42e9_999c_bb0c54522916.slice/crio-bedd16650642b7d6b5a36806c492372d943a5d993c6f21dc7c4b43d4d4f01841 WatchSource:0}: Error finding container bedd16650642b7d6b5a36806c492372d943a5d993c6f21dc7c4b43d4d4f01841: Status 404 returned error can't find the container with id bedd16650642b7d6b5a36806c492372d943a5d993c6f21dc7c4b43d4d4f01841 Jul 02 20:59:24.609323 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:24.609298 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:59:25.595701 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:25.595675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/must-gather-lg64z" event={"ID":"724903fe-099b-42e9-999c-bb0c54522916","Type":"ContainerStarted","Data":"bedd16650642b7d6b5a36806c492372d943a5d993c6f21dc7c4b43d4d4f01841"} Jul 02 20:59:26.604951 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:26.604908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/must-gather-lg64z" event={"ID":"724903fe-099b-42e9-999c-bb0c54522916","Type":"ContainerStarted","Data":"5167113a345c2b878cc33b60b728fc83d98c2d2ccd55adcd05878c0bca894c99"} Jul 02 20:59:26.605904 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:26.605875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/must-gather-lg64z" event={"ID":"724903fe-099b-42e9-999c-bb0c54522916","Type":"ContainerStarted","Data":"270562d21737532f6e2fe389efdb3aa23f05571c72e77fe8b4e3adcd2b8a5f24"} Jul 02 20:59:26.623921 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:26.623867 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9mtnv/must-gather-lg64z" podStartSLOduration=1.715641689 podStartE2EDuration="2.623816608s" totalImagesPullingTime="908.174919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:59:26.621060322 +0000 UTC m=+2674.826446953" watchObservedRunningTime="2026-07-02 20:59:26.623816608 +0000 UTC m=+2674.829203239" Jul 02 20:59:27.054620 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:27.054584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bx5qb_b6a8c413-e177-4d35-95d7-43f15b643628/global-pull-secret-syncer/0.log" Jul 02 20:59:27.214587 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:27.214553 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r5s5n_dbcad745-a614-4cca-a2b8-f36c660d9251/konnectivity-agent/0.log" Jul 02 20:59:27.315686 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:27.315603 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-106.ec2.internal_6ef5ddb8f44f9117f4e576e103d305f9/haproxy/0.log" Jul 02 20:59:32.415865 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.414612 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc4b4dc57-xpbmv_ca470f80-7437-4b4c-af5e-3c0137405e6a/authorino/0.log" Jul 02 20:59:32.442230 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.441974 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pzxrv_5bde44fc-d3a3-4236-bcc6-eea5acf608c6/manager/0.log" Jul 02 20:59:32.468487 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.468458 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qskkq_da68f9e7-997a-4697-96c7-c6983f10a42a/manager/0.log" Jul 02 20:59:32.547972 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.547927 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wc5gs_a962f7e1-3278-45bc-b5c4-a26b54d24f00/registry-server/0.log" Jul 02 20:59:32.792314 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.792007 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ncrdf_2e95eb86-62a3-4347-957c-d068324317de/manager/0.log" Jul 02 20:59:32.817412 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.816999 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fnx8p_a0d16f89-a00d-4eb8-9ac7-678858640de7/limitador/0.log" Jul 02 20:59:32.925660 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:32.925623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-65sz6_6ce73322-ad0c-49d7-b489-47ce3b7798b6/manager/0.log" Jul 02 20:59:34.424506 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.424468 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/alertmanager/0.log" Jul 02 20:59:34.450112 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.450079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/config-reloader/0.log" Jul 02 20:59:34.478118 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.478083 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/kube-rbac-proxy-web/0.log" Jul 02 20:59:34.499861 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.499273 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/kube-rbac-proxy/0.log" Jul 02 20:59:34.537332 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.537244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/kube-rbac-proxy-metric/0.log" Jul 02 20:59:34.557403 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.557323 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/prom-label-proxy/0.log" Jul 02 20:59:34.583216 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.583190 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2722554-4573-42b4-8a1d-1b75870e7196/init-config-reloader/0.log" Jul 02 20:59:34.800529 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.800488 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g555n_f46ad247-2f41-40e6-a6c1-40e05ac7323d/node-exporter/0.log" Jul 02 20:59:34.833579 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.833545 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g555n_f46ad247-2f41-40e6-a6c1-40e05ac7323d/kube-rbac-proxy/0.log" Jul 02 20:59:34.860130 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:34.859977 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g555n_f46ad247-2f41-40e6-a6c1-40e05ac7323d/init-textfile/0.log" Jul 02 20:59:35.162996 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.162959 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/prometheus/0.log" Jul 02 20:59:35.187263 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.187236 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/config-reloader/0.log" Jul 02 20:59:35.212338 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.212259 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/thanos-sidecar/0.log" Jul 02 20:59:35.253182 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.253126 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/kube-rbac-proxy-web/0.log" Jul 02 20:59:35.277352 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.277324 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/kube-rbac-proxy/0.log" Jul 02 20:59:35.299589 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.299563 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/kube-rbac-proxy-thanos/0.log" Jul 02 20:59:35.336465 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:35.336416 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a10bc5d-3c3a-4532-b4d8-d93ff115748f/init-config-reloader/0.log" Jul 02 20:59:37.733628 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.733590 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh"] Jul 02 20:59:37.738623 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.738592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.747536 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.747508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh"] Jul 02 20:59:37.794456 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.794421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-sys\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.794456 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.794459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-proc\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.794709 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.794497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6r69\" (UniqueName: \"kubernetes.io/projected/5906ee1c-6596-41de-b50f-74ebb39b9776-kube-api-access-c6r69\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.794709 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.794518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-lib-modules\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.794709 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.794548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-podres\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-sys\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896146 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-proc\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6r69\" (UniqueName: \"kubernetes.io/projected/5906ee1c-6596-41de-b50f-74ebb39b9776-kube-api-access-c6r69\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-lib-modules\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896402 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-podres\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896561 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-podres\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896561 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896519 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-sys\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.896666 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.896565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-proc\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.897030 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.897005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5906ee1c-6596-41de-b50f-74ebb39b9776-lib-modules\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:37.906653 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:37.906621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6r69\" (UniqueName: \"kubernetes.io/projected/5906ee1c-6596-41de-b50f-74ebb39b9776-kube-api-access-c6r69\") pod \"perf-node-gather-daemonset-q7bfh\" (UID: \"5906ee1c-6596-41de-b50f-74ebb39b9776\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:38.053813 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:38.053778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:38.240958 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:38.240930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh"] Jul 02 20:59:38.243236 ip-10-0-139-106 kubenswrapper[2579]: W0702 20:59:38.243205 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5906ee1c_6596_41de_b50f_74ebb39b9776.slice/crio-f349ba3a802a75cf25e45582f74ff89c8434fd7e3f5e98d1f4b58cb4ea72fbdc WatchSource:0}: Error finding container f349ba3a802a75cf25e45582f74ff89c8434fd7e3f5e98d1f4b58cb4ea72fbdc: Status 404 returned error can't find the container with id f349ba3a802a75cf25e45582f74ff89c8434fd7e3f5e98d1f4b58cb4ea72fbdc Jul 02 20:59:38.702818 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:38.702772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" event={"ID":"5906ee1c-6596-41de-b50f-74ebb39b9776","Type":"ContainerStarted","Data":"fa6ddb8d48bf8376b9af2a875d5a22bb3554a9cf28624d0806e0d7239dca0c22"} Jul 02 20:59:38.703041 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:38.702865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" event={"ID":"5906ee1c-6596-41de-b50f-74ebb39b9776","Type":"ContainerStarted","Data":"f349ba3a802a75cf25e45582f74ff89c8434fd7e3f5e98d1f4b58cb4ea72fbdc"} Jul 02 20:59:38.703041 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:38.702993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:39.825870 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:39.825817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6jvl_2120775c-f933-4bbb-911f-eca81d360290/dns/0.log" Jul 02 20:59:39.856856 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:39.856785 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6jvl_2120775c-f933-4bbb-911f-eca81d360290/kube-rbac-proxy/0.log" Jul 02 20:59:39.894043 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:39.894007 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5nfrl_c555238c-220f-4d0b-ae19-5f4fa5095ba1/dns-node-resolver/0.log" Jul 02 20:59:40.598211 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:40.598138 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6898868755-jm2z9_cd403086-32d3-42d4-bf9b-e52781fe16c0/registry/0.log" Jul 02 20:59:40.765493 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:40.765464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tlgf8_b9c8e8e5-5210-4889-9203-4d0b052a08c4/node-ca/0.log" Jul 02 20:59:41.927615 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:41.927583 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-fd64s_27691e00-296f-40e2-9c6e-09bd198c10f3/kube-auth-proxy/0.log" Jul 02 20:59:41.961594 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:41.961555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/0.log" Jul 02 20:59:41.999708 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:41.999675 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/1.log" Jul 02 20:59:42.861426 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:42.861356 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v7fwz_c287dd62-61a4-4a48-a46a-148d2728a5f3/serve-healthcheck-canary/0.log" Jul 02 20:59:43.515983 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:43.515954 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jllgd_1a68e4a9-eead-4f8a-952d-73a22eaaf7b8/kube-rbac-proxy/0.log" Jul 02 20:59:43.552656 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:43.552631 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jllgd_1a68e4a9-eead-4f8a-952d-73a22eaaf7b8/exporter/0.log" Jul 02 20:59:43.574575 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:43.574541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jllgd_1a68e4a9-eead-4f8a-952d-73a22eaaf7b8/extractor/0.log" Jul 02 20:59:44.718069 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:44.718040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" Jul 02 20:59:44.738041 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:44.737984 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-q7bfh" podStartSLOduration=7.737963607 podStartE2EDuration="7.737963607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:59:38.721582785 +0000 UTC m=+2686.926969440" watchObservedRunningTime="2026-07-02 20:59:44.737963607 +0000 UTC m=+2692.943350238" Jul 02 20:59:45.707301 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:45.707264 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r7b44_d4056c24-33f2-48d9-813f-5b001e4b9e35/manager/0.log" Jul 02 20:59:45.883502 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:45.883459 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-th9ph_f41800cf-ceac-4609-a972-d459ea3ea166/manager/0.log" Jul 02 20:59:46.088635 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:46.088555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ckzps_fc4ea332-04e9-4127-9483-8196f79f50e7/server/0.log" Jul 02 20:59:46.168866 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:46.168812 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4mk8s_c35bd262-954b-4fa0-9981-dccf2efedd71/manager/0.log" Jul 02 20:59:46.388032 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:46.387983 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-ksc2n_11751f38-8ff4-4712-9f3a-36f70e815271/manager/0.log" Jul 02 20:59:46.405562 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:46.405525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-ksc2n_11751f38-8ff4-4712-9f3a-36f70e815271/copy-manifests/0.log" Jul 02 20:59:48.086028 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:48.085998 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-fd64s_27691e00-296f-40e2-9c6e-09bd198c10f3/kube-auth-proxy/0.log" Jul 02 20:59:48.104366 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:48.104165 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/0.log" Jul 02 20:59:48.136689 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:48.136594 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/1.log" Jul 02 20:59:49.544140 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:49.544101 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-fd64s_27691e00-296f-40e2-9c6e-09bd198c10f3/kube-auth-proxy/0.log" Jul 02 20:59:49.563856 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:49.563809 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/0.log" Jul 02 20:59:49.600422 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:49.600387 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-k5kdw_b805a60f-d88c-4b41-80a0-05e23ccb1451/istio-proxy/1.log" Jul 02 20:59:54.830985 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.830909 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/kube-multus-additional-cni-plugins/0.log" Jul 02 20:59:54.852690 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.852658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/egress-router-binary-copy/0.log" Jul 02 20:59:54.878319 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.878283 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/cni-plugins/0.log" Jul 02 20:59:54.903630 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.903596 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/bond-cni-plugin/0.log" Jul 02 20:59:54.926567 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.926524 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/routeoverride-cni/0.log" Jul 02 20:59:54.947370 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.947337 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/whereabouts-cni-bincopy/0.log" Jul 02 20:59:54.968204 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:54.968174 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s4hd7_7c032036-fc2d-4dc2-a98c-6bc734d95c38/whereabouts-cni/0.log" Jul 02 20:59:55.068590 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:55.068540 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wvsc6_60ae01ac-7f49-40e6-a2a7-7faabd755c21/kube-multus/0.log" Jul 02 20:59:55.095976 ip-10-0-139-106 kubenswrapper[2579]: I0702 20:59:55.095895 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8dm9k_c1340cfc-2d2c-4048-af47-f70bc4fa1bb2/network-metrics-daemon/0.log"