Jul 06 19:49:10.613945 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:49:10.613957 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:49:10.613965 ip-10-0-140-110 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:49:10.614192 ip-10-0-140-110 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:49:20.653042 ip-10-0-140-110 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:49:20.653061 ip-10-0-140-110 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 942e0e7cdb2140f7ad13ffea5926e640 -- Jul 06 19:51:15.497610 ip-10-0-140-110 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:51:15.984413 ip-10-0-140-110 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.908660 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912082 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912095 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912099 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912102 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912105 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912109 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912111 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.008725 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912114 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912117 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912120 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912124 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912126 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912129 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912132 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912135 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912139 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912143 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912146 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912148 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912151 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912156 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912159 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912162 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912166 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912171 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.009686 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912176 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912178 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912181 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912184 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912187 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912190 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912192 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912195 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912197 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912200 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912203 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912206 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912208 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912211 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912214 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912216 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912219 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912221 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912224 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912226 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.010287 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912229 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912231 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912234 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912236 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912239 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912242 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912245 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912248 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912250 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912253 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912255 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912258 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912260 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912263 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912265 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912268 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912270 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912273 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912275 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912278 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.010983 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912280 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912283 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912285 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912288 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912290 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912293 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912296 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912298 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912301 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912303 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912306 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912308 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912310 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912313 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912316 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912318 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912336 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912341 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912344 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912347 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.011881 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912350 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912353 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912356 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912358 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912361 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912364 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912367 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912370 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912372 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912375 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912377 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912380 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912383 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.912385 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913104 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913111 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913115 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913118 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913121 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913125 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.012793 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913128 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913131 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913134 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913136 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913139 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913141 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913144 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913148 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913151 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913154 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913157 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913160 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913162 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913165 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913167 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913170 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913172 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913175 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913179 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.013370 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913182 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913185 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913188 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913190 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913193 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913196 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913199 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913202 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913204 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913207 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913209 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913212 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913215 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913218 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913220 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913223 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913226 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913229 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913231 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913234 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.014076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913236 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913239 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913241 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913243 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913246 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913248 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913251 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913253 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913256 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913258 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913263 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913266 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913269 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913271 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913274 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913276 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913279 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913281 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913284 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913286 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.014739 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913289 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913292 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913294 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913297 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913299 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913301 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913304 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913307 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913309 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913314 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913317 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913337 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913340 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913343 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913345 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913348 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913352 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913354 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913357 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.015404 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913359 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913362 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913365 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913369 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913373 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913375 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913378 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913380 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913383 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913385 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913388 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913390 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913393 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913396 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913398 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913775 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913780 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913783 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913786 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913789 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.016282 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913792 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913795 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913798 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913800 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913803 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913805 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913808 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913810 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913813 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913817 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913820 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913822 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913825 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913827 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913830 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913834 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913837 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913840 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913843 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913845 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.017121 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913848 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913852 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913856 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913859 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913862 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913865 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913867 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913870 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913873 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913875 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913877 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913880 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913883 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913885 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913887 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913890 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913893 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913895 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913897 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.017951 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913900 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913902 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913907 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913910 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913913 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913916 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913919 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913921 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913925 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913928 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913931 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913933 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913936 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913938 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913941 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913943 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913946 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913949 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913951 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.018846 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913954 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913956 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913958 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913961 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913963 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913966 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913969 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913972 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913975 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913977 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913980 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913983 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913987 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913991 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913993 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.913998 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914001 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914005 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914008 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914011 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.019749 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914013 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914017 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914019 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914022 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914024 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914027 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914029 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914032 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914034 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914036 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914039 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914041 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914044 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914046 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914049 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914051 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914119 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914130 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914145 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914154 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914158 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:51:16.020587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914162 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914166 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914170 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914173 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914176 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914179 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914184 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914187 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914190 2564 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914194 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914197 2564 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914200 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914204 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914208 2564 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914211 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914214 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914217 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914221 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914225 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914228 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914231 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914234 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914237 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914240 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:51:16.021239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914243 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914246 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914249 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914253 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914256 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914259 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914262 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914265 2564 flags.go:64] FLAG: --enable-server="true" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914269 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914291 2564 flags.go:64] FLAG: --event-burst="100" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914295 2564 flags.go:64] FLAG: --event-qps="50" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914298 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914301 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914304 2564 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914309 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914312 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914315 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914318 2564 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914335 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914338 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914342 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914345 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914348 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914351 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914354 2564 flags.go:64] FLAG: --feature-gates="" Jul 06 19:51:16.021909 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914358 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914361 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914364 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914367 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914371 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914374 2564 flags.go:64] FLAG: --help="false" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914377 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914380 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914383 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914386 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914389 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914392 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914395 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914398 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914401 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914403 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914407 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914410 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914413 2564 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914416 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914419 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914423 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914426 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914429 2564 flags.go:64] FLAG: --lock-file="" Jul 06 19:51:16.024869 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914431 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914434 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914437 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914444 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914447 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914450 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914452 2564 flags.go:64] FLAG: --logging-format="text" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914455 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914459 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914462 2564 flags.go:64] FLAG: --manifest-url="" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914465 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914469 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914472 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914476 2564 flags.go:64] FLAG: --max-pods="110" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914479 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914482 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914485 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914488 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914491 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914494 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914497 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914504 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914506 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914509 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:51:16.025716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914512 2564 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914515 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914521 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914524 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914527 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914530 2564 flags.go:64] FLAG: --port="10250" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914534 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914537 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fb6a7192ff5bb7b0" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914541 2564 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914544 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914547 2564 flags.go:64] FLAG: --register-node="true" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914551 2564 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914554 2564 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914557 2564 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914560 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914563 2564 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914571 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914574 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914577 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914580 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914583 2564 flags.go:64] FLAG: --runonce="false" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914586 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914589 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914592 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914595 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:51:16.136777 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914598 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914601 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914604 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914607 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914610 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914613 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914616 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914619 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914622 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914625 2564 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914628 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914633 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914636 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914640 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914644 2564 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914648 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914650 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914653 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914658 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914661 2564 flags.go:64] FLAG: --v="2" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914665 2564 flags.go:64] FLAG: --version="false" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914669 2564 flags.go:64] FLAG: --vmodule="" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914673 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.914676 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914787 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.150966 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914791 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914794 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914797 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914800 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914802 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914805 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914807 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914810 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914813 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914815 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914818 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914820 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914822 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914825 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914828 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914830 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914832 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914835 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914838 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914840 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.151762 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914844 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914847 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914849 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914852 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914855 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914859 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914861 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914864 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914866 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914869 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914871 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914874 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914877 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914879 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914882 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914884 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914887 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914889 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914892 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914894 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.152476 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914896 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914899 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914901 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914904 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914906 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914910 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914912 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914914 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914917 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914920 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914923 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914926 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914929 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914932 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914935 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914939 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914942 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914946 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914949 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.153070 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914952 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914954 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914958 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914962 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914966 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914969 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914972 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914974 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914977 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914980 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914982 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914984 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914987 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914989 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914992 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914994 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914997 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.914999 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915002 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.153698 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915004 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915007 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915010 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915012 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915015 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915017 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915021 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915024 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915026 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915029 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915031 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915038 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915040 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915043 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915045 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915048 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915051 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915053 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915056 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.154242 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.915058 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.916117 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.923474 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.923488 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.923501 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.923511 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923569 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923573 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923577 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923579 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923583 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923586 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923589 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923591 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923594 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.154825 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923597 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923599 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923602 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923605 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923607 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923612 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923615 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923617 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923620 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923626 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923629 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923633 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923637 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923641 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923645 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923648 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923651 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923654 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923656 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.155261 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923659 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923663 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923665 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923668 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923670 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923673 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923675 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923678 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923681 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923683 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923686 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923688 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923691 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923694 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923696 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923699 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923701 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923704 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923708 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923711 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.155866 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923713 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923716 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923720 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923722 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923725 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923727 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923730 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923733 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923735 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923738 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923740 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923743 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923745 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923748 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923751 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923754 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923756 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923759 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923762 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923764 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.156488 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923767 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923769 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923772 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923774 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923776 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923779 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923781 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923784 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923786 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923789 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923792 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923795 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923797 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923804 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923806 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923809 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923811 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923814 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.157076 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923817 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923819 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923822 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923824 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923827 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923829 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923832 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923835 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923837 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923840 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923844 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923848 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.923853 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923967 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:51:16.157851 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923971 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923974 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923977 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923980 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923982 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923985 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923987 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923990 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923993 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923995 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.923999 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924002 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924005 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924009 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924011 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924014 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924016 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924019 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924022 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924024 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:51:16.158732 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924027 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924029 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924032 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924034 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924037 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924039 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924042 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924045 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924048 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924050 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924053 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924056 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924058 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924060 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924063 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924066 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924068 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924070 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924073 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924075 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:51:16.159595 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924078 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924080 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924084 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924086 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924089 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924093 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924095 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924098 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924101 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924103 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924106 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924108 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924111 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924113 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924115 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924118 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924120 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924123 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924126 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:51:16.160232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924129 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924136 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924139 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924142 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924144 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924147 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924149 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924152 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924154 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924156 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924159 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924161 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924164 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924167 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924169 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924175 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924177 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924180 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924184 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924186 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:51:16.161075 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924189 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924191 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924194 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924196 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924199 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924202 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924204 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924207 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924209 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924213 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924216 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924219 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924222 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924225 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924228 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924231 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924233 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924237 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:51:16.161935 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:15.924240 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.924245 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.924365 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.928476 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.929703 2564 server.go:1019] "Starting client certificate rotation" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.929799 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.929827 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.932127 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.933443 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.949507 2564 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.951771 2564 log.go:25] "Validated CRI v1 image API" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.952468 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:51:16.162561 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.954918 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a77227d2-a702-4f1d-a032-382f4cb77c69:/dev/nvme0n1p4 f8f82552-1000-4687-a2cd-22450fd95704:/dev/nvme0n1p3] Jul 06 19:51:16.163189 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.954933 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.960400 2564 manager.go:216] Machine: {Timestamp:2026-07-06 19:51:15.959235543 +0000 UTC m=+0.344772419 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098111 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fcd6b6537e6386508a189af65255b SystemUUID:ec2fcd6b-6537-e638-6508-a189af65255b BootID:942e0e7c-db21-40f7-ad13-ffea5926e640 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:4c:60:4c:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:4c:60:4c:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:84:dd:c4:ee:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.960497 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.960576 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.963765 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.963797 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.964793 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.964803 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:51:16.165521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.964816 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.965411 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.966788 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969629 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969737 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969770 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969790 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969806 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969815 2564 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969825 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.969851 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.970136 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.970151 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.973469 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.976147 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.976174 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977440 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977459 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977465 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977471 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977480 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977493 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977501 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977507 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977514 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977520 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977530 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977536 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977567 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.977574 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.981519 2564 server.go:1268] "Started kubelet" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.981533 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.981611 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.981669 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.982771 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.986370 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.988170 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:15.989306 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:15.989309 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:51:16.165944 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:15.989872 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.990582 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.990590 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:15.990650 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991135 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991151 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991164 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991252 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991260 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991488 2564 factory.go:55] Registering systemd factory Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991504 2564 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991676 2564 factory.go:153] Registering CRI-O factory Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991688 2564 factory.go:223] Registration of the crio container factory successfully Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991740 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991758 2564 factory.go:103] Registering Raw factory Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.991769 2564 manager.go:1196] Started watching for new ooms in manager Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:15.992596 2564 manager.go:349] Starting recovery of all containers Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:15.991296 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18bfcb454dd43f10 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:ip-10-0-140-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-07-06 19:51:15.98094312 +0000 UTC m=+0.366479975,LastTimestamp:2026-07-06 19:51:15.98094312 +0000 UTC m=+0.366479975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.001519 2564 manager.go:354] Recovery completed Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.002575 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.002698 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.005430 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qm48j" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.007579 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011295 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011339 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011351 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011957 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011969 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.011988 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.012021 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.013300 2564 policy_none.go:49] "None policy: Start" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.013336 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.013353 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.014491 2564 policy_none.go:47] "Start" Jul 06 19:51:16.167428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.016174 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qm48j" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.016154 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-110.ec2.internal.18bfcb454fa3918b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-110.ec2.internal,UID:ip-10-0-140-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-110.ec2.internal,},FirstTimestamp:2026-07-06 19:51:16.011307403 +0000 UTC m=+0.396844257,LastTimestamp:2026-07-06 19:51:16.011307403 +0000 UTC m=+0.396844257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-110.ec2.internal,}" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.027524 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.027559 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.027585 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061011 2564 manager.go:337] "Starting Device Plugin manager" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.061222 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061235 2564 server.go:87] "Starting device plugin registration server" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061473 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061484 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061593 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061659 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.061668 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.070832 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.070880 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.160442 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.161661 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.161793 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.161838 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.161859 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.161898 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.165265 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.167741 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.167773 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.167787 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.168547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.167810 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.183831 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.183849 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-110.ec2.internal\": node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.255264 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.262527 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"] Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.262583 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.264113 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.264142 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.264152 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.265484 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.265609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.265639 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266163 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266190 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266163 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266225 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266237 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.266201 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.267487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.267508 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.268135 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.268152 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.268161 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.292002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.292029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:16.295090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:16.292053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.304931 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:16.310961 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-110.ec2.internal\" not found" node="ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:17.900319 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936848 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/813e38510b3ec379e921946926d00ffe-config\") pod \"kube-apiserver-proxy-ip-10-0-140-110.ec2.internal\" (UID: \"813e38510b3ec379e921946926d00ffe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936872 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:17.954636 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:17.936898 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f44b43a3e849fcf8ea37cb5848fb430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal\" (UID: \"7f44b43a3e849fcf8ea37cb5848fb430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:18.098926 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.001332 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:18.205167 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.102426 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:18.205167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.151899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:18.205167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.158795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:18.205167 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.203185 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:18.364893 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.303661 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.404120 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-110.ec2.internal\" not found" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.457597 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.473995 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.474104 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.474115 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:51:18.523774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.517064 2564 apiserver.go:52] "Watching apiserver" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.535223 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.535241 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.537463 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.537754 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-t5w56","kube-system/konnectivity-agent-vflbj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db","openshift-image-registry/node-ca-s8jgg","openshift-multus/multus-additional-cni-plugins-5bdqd","openshift-network-operator/iptables-alerter-j6x5z","openshift-ovn-kubernetes/ovnkube-node-qb4xh","openshift-cluster-node-tuning-operator/tuned-6dsx6","openshift-dns/node-resolver-5q9w2","openshift-multus/multus-rrcnv","openshift-multus/network-metrics-daemon-cq5hm","openshift-network-diagnostics/network-check-target-zlngt"] Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.539092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.539147 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.542000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.542002 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.542533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.543796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.545042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.546237 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.546354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.546501 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.547601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.548340 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-82b8z\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.548599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.549770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.551046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.551111 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552556 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6mvs\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552610 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552675 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552700 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.625602 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552909 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.552983 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553029 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553143 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553177 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553366 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.553381 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553425 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553617 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553751 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.553912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.554055 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.557323 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.558478 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal"] Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.558530 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.558565 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559085 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6xgz4\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559132 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559158 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559735 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pfrxc\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559747 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559781 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mvrlt\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dcd22\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77jz6\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.559736 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.560144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z2q86\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.560355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:51:18.627546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.563328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.563481 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.563721 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.564418 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2js59\"" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.564810 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.565600 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:46:17 +0000 UTC" deadline="2028-03-18 09:25:38.26428908 +0000 UTC" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.565641 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14893h34m19.698652288s" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.574029 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal"] Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.574145 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.588363 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rlvg7" Jul 06 19:51:18.629316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.604628 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rlvg7" Jul 06 19:51:18.716054 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.640678 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-os-release\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b803583c-da0c-4429-826d-0e29a5109686-host-slash\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740224 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-kubelet\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-var-lib-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e347eae-c86f-4c0c-a8d5-d191b34e1048-host\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/0e347eae-c86f-4c0c-a8d5-d191b34e1048-kube-api-access-d58hf\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-kubernetes\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b803583c-da0c-4429-826d-0e29a5109686-iptables-alerter-script\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-env-overrides\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysconfig\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-kubelet\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-konnectivity-ca\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:18.832747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740575 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-netns\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-conf\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-socket-dir-parent\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-k8s-cni-cncf-io\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqdh\" (UniqueName: \"kubernetes.io/projected/eb4207db-8056-491e-9d95-87a6b95e9dd0-kube-api-access-jnqdh\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-cnibin\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-ovn\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-script-lib\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/603d4e89-5bc7-4145-8fef-85bf2976fc6b-hosts-file\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d4e89-5bc7-4145-8fef-85bf2976fc6b-tmp-dir\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-cnibin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.833321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.740995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clmb2\" (UniqueName: \"kubernetes.io/projected/b803583c-da0c-4429-826d-0e29a5109686-kube-api-access-clmb2\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-socket-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-run\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-var-lib-kubelet\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-system-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-netns\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-systemd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-log-socket\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-systemd\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-os-release\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-conf-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ws67\" (UniqueName: \"kubernetes.io/projected/222059e5-18ec-46b0-8613-d310fc61925b-kube-api-access-8ws67\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.834305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-node-log\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.741997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-registration-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-bin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-multus\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-hostroot\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-netd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfk5\" (UniqueName: \"kubernetes.io/projected/86913132-0316-44d0-a828-eaf00b45c59c-kube-api-access-9vfk5\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-tuned\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-tmp\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e347eae-c86f-4c0c-a8d5-d191b34e1048-serviceca\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:18.834774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-sys\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-lib-modules\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-agent-certs\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr67z\" (UniqueName: \"kubernetes.io/projected/f57894ae-5abb-42ec-b9d1-5e4cd912760d-kube-api-access-lr67z\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzmx\" (UniqueName: \"kubernetes.io/projected/933557bc-75c1-414b-841b-df6c6b75c32e-kube-api-access-kbzmx\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-cni-binary-copy\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-etc-kubernetes\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-systemd-units\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-etc-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-kubelet-config\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:18.835268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.742989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-dbus\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-device-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-etc-selinux\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-config\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovn-node-metrics-cert\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-modprobe-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-host\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhv9s\" (UniqueName: \"kubernetes.io/projected/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-kube-api-access-qhv9s\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-daemon-config\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-slash\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-bin\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-multus-certs\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-sys-fs\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.835838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.743616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxth\" (UniqueName: \"kubernetes.io/projected/603d4e89-5bc7-4145-8fef-85bf2976fc6b-kube-api-access-8rxth\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:18.836325 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:18.809220 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f44b43a3e849fcf8ea37cb5848fb430.slice/crio-646e546478b5076aca625fdbb7b44dbe2d875bd16ffb5255a8e6e0b7a5ce8916 WatchSource:0}: Error finding container 646e546478b5076aca625fdbb7b44dbe2d875bd16ffb5255a8e6e0b7a5ce8916: Status 404 returned error can't find the container with id 646e546478b5076aca625fdbb7b44dbe2d875bd16ffb5255a8e6e0b7a5ce8916 Jul 06 19:51:18.836325 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:18.809512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813e38510b3ec379e921946926d00ffe.slice/crio-6b777c1a8f82c265ed2d4356a3727abc03129e4fc9e732a1aba6d6215c45687b WatchSource:0}: Error finding container 6b777c1a8f82c265ed2d4356a3727abc03129e4fc9e732a1aba6d6215c45687b: Status 404 returned error can't find the container with id 6b777c1a8f82c265ed2d4356a3727abc03129e4fc9e732a1aba6d6215c45687b Jul 06 19:51:18.836325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.816907 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d4e89-5bc7-4145-8fef-85bf2976fc6b-tmp-dir\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-cnibin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clmb2\" (UniqueName: \"kubernetes.io/projected/b803583c-da0c-4429-826d-0e29a5109686-kube-api-access-clmb2\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-socket-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-run\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-var-lib-kubelet\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-system-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-netns\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844781 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-cnibin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844837 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-var-lib-kubelet\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844870 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844891 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-system-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844891 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-socket-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:18.932750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844923 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-netns\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-systemd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844954 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-log-socket\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.844986 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-systemd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-systemd\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-log-socket\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-os-release\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-run\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845039 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/603d4e89-5bc7-4145-8fef-85bf2976fc6b-tmp-dir\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-conf-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-systemd\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-os-release\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ws67\" (UniqueName: \"kubernetes.io/projected/222059e5-18ec-46b0-8613-d310fc61925b-kube-api-access-8ws67\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.061311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-node-log\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-registration-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845091 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-conf-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845146 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-node-log\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-bin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-multus\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845196 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-registration-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-hostroot\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845230 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-multus\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845245 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-cni-bin\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845270 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-hostroot\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-netd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfk5\" (UniqueName: \"kubernetes.io/projected/86913132-0316-44d0-a828-eaf00b45c59c-kube-api-access-9vfk5\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845336 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-netd\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.063639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-tuned\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-tmp\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e347eae-c86f-4c0c-a8d5-d191b34e1048-serviceca\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845460 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-sys\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-lib-modules\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-agent-certs\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr67z\" (UniqueName: \"kubernetes.io/projected/f57894ae-5abb-42ec-b9d1-5e4cd912760d-kube-api-access-lr67z\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzmx\" (UniqueName: \"kubernetes.io/projected/933557bc-75c1-414b-841b-df6c6b75c32e-kube-api-access-kbzmx\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-cni-binary-copy\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.064487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-etc-kubernetes\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845717 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.845752 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.845795 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:19.345779284 +0000 UTC m=+2.186781736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-systemd-units\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-etc-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-kubelet-config\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-dbus\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-device-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-etc-selinux\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.065623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.845986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-config\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovn-node-metrics-cert\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-modprobe-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-host\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhv9s\" (UniqueName: \"kubernetes.io/projected/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-kube-api-access-qhv9s\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846105 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e347eae-c86f-4c0c-a8d5-d191b34e1048-serviceca\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-daemon-config\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-slash\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846144 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-bin\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.846184 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846193 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-sys\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846220 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-multus-certs\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.846238 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:19.346222864 +0000 UTC m=+2.187225315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:19.066233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-multus-certs\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846258 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-systemd-units\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-sys-fs\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxth\" (UniqueName: \"kubernetes.io/projected/603d4e89-5bc7-4145-8fef-85bf2976fc6b-kube-api-access-8rxth\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846291 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-etc-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-os-release\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b803583c-da0c-4429-826d-0e29a5109686-host-slash\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-kubelet\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846380 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-kubelet-config\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-var-lib-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e347eae-c86f-4c0c-a8d5-d191b34e1048-host\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/0e347eae-c86f-4c0c-a8d5-d191b34e1048-kube-api-access-d58hf\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-kubernetes\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846472 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e613ae5-5cda-463a-8a25-3b357a21de22-dbus\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.067322 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-device-dir\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-sys-fs\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-cni-dir\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846629 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/933557bc-75c1-414b-841b-df6c6b75c32e-etc-selinux\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-os-release\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-lib-modules\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846791 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b803583c-da0c-4429-826d-0e29a5109686-host-slash\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846835 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-kubelet\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846894 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-var-lib-openvswitch\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.846932 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e347eae-c86f-4c0c-a8d5-d191b34e1048-host\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847043 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/222059e5-18ec-46b0-8613-d310fc61925b-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-cni-binary-copy\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-kubernetes\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847163 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-etc-kubernetes\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.068207 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847251 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-modprobe-d\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847347 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-host\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847395 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-slash\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847467 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-cni-bin\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b803583c-da0c-4429-826d-0e29a5109686-iptables-alerter-script\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-env-overrides\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847548 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-config\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysconfig\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-kubelet\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-konnectivity-ca\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-netns\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-conf\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-socket-dir-parent\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.069518 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-k8s-cni-cncf-io\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqdh\" (UniqueName: \"kubernetes.io/projected/eb4207db-8056-491e-9d95-87a6b95e9dd0-kube-api-access-jnqdh\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-cnibin\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-ovn\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847918 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-env-overrides\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-daemon-config\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-script-lib\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.847964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/603d4e89-5bc7-4145-8fef-85bf2976fc6b-hosts-file\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848008 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-multus-socket-dir-parent\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/603d4e89-5bc7-4145-8fef-85bf2976fc6b-hosts-file\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysconfig\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848104 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.070700 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848110 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-var-lib-kubelet\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848125 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-cnibin\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848150 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/222059e5-18ec-46b0-8613-d310fc61925b-system-cni-dir\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848164 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb4207db-8056-491e-9d95-87a6b95e9dd0-host-run-k8s-cni-cncf-io\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848186 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-run-ovn\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848355 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f57894ae-5abb-42ec-b9d1-5e4cd912760d-host-run-netns\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b803583c-da0c-4429-826d-0e29a5109686-iptables-alerter-script\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848495 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-sysctl-conf\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.848591 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovnkube-script-lib\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.849256 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-konnectivity-ca\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.849361 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-tmp\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.849392 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-etc-tuned\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.849634 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7-agent-certs\") pod \"konnectivity-agent-vflbj\" (UID: \"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7\") " pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.850274 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f57894ae-5abb-42ec-b9d1-5e4cd912760d-ovn-node-metrics-cert\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071167 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.882043 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.886531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ws67\" (UniqueName: \"kubernetes.io/projected/222059e5-18ec-46b0-8613-d310fc61925b-kube-api-access-8ws67\") pod \"multus-additional-cni-plugins-5bdqd\" (UID: \"222059e5-18ec-46b0-8613-d310fc61925b\") " pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.889919 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfk5\" (UniqueName: \"kubernetes.io/projected/86913132-0316-44d0-a828-eaf00b45c59c-kube-api-access-9vfk5\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.890251 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.890271 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.890285 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.890284 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/0e347eae-c86f-4c0c-a8d5-d191b34e1048-kube-api-access-d58hf\") pod \"node-ca-s8jgg\" (UID: \"0e347eae-c86f-4c0c-a8d5-d191b34e1048\") " pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:18.890338 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:19.390320359 +0000 UTC m=+2.231322818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.890665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr67z\" (UniqueName: \"kubernetes.io/projected/f57894ae-5abb-42ec-b9d1-5e4cd912760d-kube-api-access-lr67z\") pod \"ovnkube-node-qb4xh\" (UID: \"f57894ae-5abb-42ec-b9d1-5e4cd912760d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.892197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxth\" (UniqueName: \"kubernetes.io/projected/603d4e89-5bc7-4145-8fef-85bf2976fc6b-kube-api-access-8rxth\") pod \"node-resolver-5q9w2\" (UID: \"603d4e89-5bc7-4145-8fef-85bf2976fc6b\") " pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.892528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clmb2\" (UniqueName: \"kubernetes.io/projected/b803583c-da0c-4429-826d-0e29a5109686-kube-api-access-clmb2\") pod \"iptables-alerter-j6x5z\" (UID: \"b803583c-da0c-4429-826d-0e29a5109686\") " pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.893466 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhv9s\" (UniqueName: \"kubernetes.io/projected/aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb-kube-api-access-qhv9s\") pod \"tuned-6dsx6\" (UID: \"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:18.896352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aa3bdd_a39d_453c_ad92_e1f0f31f41e7.slice/crio-a904d0cf43500cbb523afb4977b93e25f6a80ad428b23f363027019cf05be6b7 WatchSource:0}: Error finding container a904d0cf43500cbb523afb4977b93e25f6a80ad428b23f363027019cf05be6b7: Status 404 returned error can't find the container with id a904d0cf43500cbb523afb4977b93e25f6a80ad428b23f363027019cf05be6b7 Jul 06 19:51:19.071644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.900964 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqdh\" (UniqueName: \"kubernetes.io/projected/eb4207db-8056-491e-9d95-87a6b95e9dd0-kube-api-access-jnqdh\") pod \"multus-rrcnv\" (UID: \"eb4207db-8056-491e-9d95-87a6b95e9dd0\") " pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.902474 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzmx\" (UniqueName: \"kubernetes.io/projected/933557bc-75c1-414b-841b-df6c6b75c32e-kube-api-access-kbzmx\") pod \"aws-ebs-csi-driver-node-cx2db\" (UID: \"933557bc-75c1-414b-841b-df6c6b75c32e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.934830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s8jgg" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:18.948652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:18.957044 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222059e5_18ec_46b0_8613_d310fc61925b.slice/crio-8f24cc05fe7a17b5e5413f2107cfafad5246b23ec81402036837fed87e65d80b WatchSource:0}: Error finding container 8f24cc05fe7a17b5e5413f2107cfafad5246b23ec81402036837fed87e65d80b: Status 404 returned error can't find the container with id 8f24cc05fe7a17b5e5413f2107cfafad5246b23ec81402036837fed87e65d80b Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.003826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6x5z" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:19.011160 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb803583c_da0c_4429_826d_0e29a5109686.slice/crio-0a54f6047ccc3813506768c30ee1eabf50288307afa84c54ef7741e6b8e9881e WatchSource:0}: Error finding container 0a54f6047ccc3813506768c30ee1eabf50288307afa84c54ef7741e6b8e9881e: Status 404 returned error can't find the container with id 0a54f6047ccc3813506768c30ee1eabf50288307afa84c54ef7741e6b8e9881e Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.019659 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:19.032983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57894ae_5abb_42ec_b9d1_5e4cd912760d.slice/crio-0a5ac92302b57d3eba0094fbfe73937180e99e4cd93b618f1baf8fa30da0d934 WatchSource:0}: Error finding container 0a5ac92302b57d3eba0094fbfe73937180e99e4cd93b618f1baf8fa30da0d934: Status 404 returned error can't find the container with id 0a5ac92302b57d3eba0094fbfe73937180e99e4cd93b618f1baf8fa30da0d934 Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.037916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:19.045032 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf34c4f_6dc4_4a61_ab2a_9af3f60b78cb.slice/crio-fd71963c6f31c2fbadc8c11d07375e8fa54cd45fca9dc67e23a69ea98a7ab602 WatchSource:0}: Error finding container fd71963c6f31c2fbadc8c11d07375e8fa54cd45fca9dc67e23a69ea98a7ab602: Status 404 returned error can't find the container with id fd71963c6f31c2fbadc8c11d07375e8fa54cd45fca9dc67e23a69ea98a7ab602 Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.061596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5q9w2" Jul 06 19:51:19.072239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.066450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrcnv" Jul 06 19:51:19.077144 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:19.077125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4207db_8056_491e_9d95_87a6b95e9dd0.slice/crio-f8e35826cded4ea617d2bf4235019418c902ad227cf33e7299fffa72de0e4675 WatchSource:0}: Error finding container f8e35826cded4ea617d2bf4235019418c902ad227cf33e7299fffa72de0e4675: Status 404 returned error can't find the container with id f8e35826cded4ea617d2bf4235019418c902ad227cf33e7299fffa72de0e4675 Jul 06 19:51:19.116559 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.116516 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:19.169532 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.169506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" Jul 06 19:51:19.177989 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:19.177970 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933557bc_75c1_414b_841b_df6c6b75c32e.slice/crio-c31a6b0b1b92d5630cc799e5e6f4ae90d6f9cb63aec75606d76c39fb2a1b3473 WatchSource:0}: Error finding container c31a6b0b1b92d5630cc799e5e6f4ae90d6f9cb63aec75606d76c39fb2a1b3473: Status 404 returned error can't find the container with id c31a6b0b1b92d5630cc799e5e6f4ae90d6f9cb63aec75606d76c39fb2a1b3473 Jul 06 19:51:19.351359 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.350979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:19.351359 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.351027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:19.351359 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.351143 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:19.351359 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.351198 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:20.351181584 +0000 UTC m=+3.192184052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:19.351707 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.351574 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:19.351707 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.351622 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:20.351607453 +0000 UTC m=+3.192609921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:19.451694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.451606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:19.451837 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.451769 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:19.451837 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.451788 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:19.451837 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.451799 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:19.452023 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.451853 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:20.45183444 +0000 UTC m=+3.292836893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:19.606277 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.606153 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:46:18 +0000 UTC" deadline="2027-12-07 22:15:25.873300559 +0000 UTC" Jul 06 19:51:19.606277 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.606221 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12458h24m6.267084239s" Jul 06 19:51:19.707648 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.707483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:19.707648 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:19.707607 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:19.724274 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.724223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrcnv" event={"ID":"eb4207db-8056-491e-9d95-87a6b95e9dd0","Type":"ContainerStarted","Data":"f8e35826cded4ea617d2bf4235019418c902ad227cf33e7299fffa72de0e4675"} Jul 06 19:51:19.746546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.746514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5q9w2" event={"ID":"603d4e89-5bc7-4145-8fef-85bf2976fc6b","Type":"ContainerStarted","Data":"95f68f50b0e432c077e77a88450f44d8c179332e227f66e3c4eccd3ce9d6e3df"} Jul 06 19:51:19.775100 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.775065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" event={"ID":"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb","Type":"ContainerStarted","Data":"fd71963c6f31c2fbadc8c11d07375e8fa54cd45fca9dc67e23a69ea98a7ab602"} Jul 06 19:51:19.809554 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.809449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"0a5ac92302b57d3eba0094fbfe73937180e99e4cd93b618f1baf8fa30da0d934"} Jul 06 19:51:19.828977 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.828823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6x5z" event={"ID":"b803583c-da0c-4429-826d-0e29a5109686","Type":"ContainerStarted","Data":"0a54f6047ccc3813506768c30ee1eabf50288307afa84c54ef7741e6b8e9881e"} Jul 06 19:51:19.866314 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.866278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s8jgg" event={"ID":"0e347eae-c86f-4c0c-a8d5-d191b34e1048","Type":"ContainerStarted","Data":"8a55e32dd3392173cd09e690a9ffa7ca59bbcb7c1d0a2d239090d21084c499b3"} Jul 06 19:51:19.876608 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.876578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vflbj" event={"ID":"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7","Type":"ContainerStarted","Data":"a904d0cf43500cbb523afb4977b93e25f6a80ad428b23f363027019cf05be6b7"} Jul 06 19:51:19.917613 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.917574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"813e38510b3ec379e921946926d00ffe","Type":"ContainerStarted","Data":"6b777c1a8f82c265ed2d4356a3727abc03129e4fc9e732a1aba6d6215c45687b"} Jul 06 19:51:19.939146 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.939113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" event={"ID":"933557bc-75c1-414b-841b-df6c6b75c32e","Type":"ContainerStarted","Data":"c31a6b0b1b92d5630cc799e5e6f4ae90d6f9cb63aec75606d76c39fb2a1b3473"} Jul 06 19:51:19.959350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.959265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerStarted","Data":"8f24cc05fe7a17b5e5413f2107cfafad5246b23ec81402036837fed87e65d80b"} Jul 06 19:51:19.992641 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:19.992608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerStarted","Data":"646e546478b5076aca625fdbb7b44dbe2d875bd16ffb5255a8e6e0b7a5ce8916"} Jul 06 19:51:20.065528 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.065343 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.358413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.358458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.358590 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.358648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:22.358631689 +0000 UTC m=+5.199634142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.359037 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:20.359208 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.359091 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:22.359074951 +0000 UTC m=+5.200077406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:20.459680 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.459636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:20.459980 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.459801 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:20.459980 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.459818 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:20.459980 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.459830 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:20.459980 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.459901 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:22.459883957 +0000 UTC m=+5.300886411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:20.607293 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.607253 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:46:18 +0000 UTC" deadline="2028-04-22 16:49:16.480720438 +0000 UTC" Jul 06 19:51:20.607293 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.607292 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15740h57m55.873432862s" Jul 06 19:51:20.707714 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.706981 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:20.707714 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.707123 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:20.707714 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.707524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:20.707714 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:20.707628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:20.986890 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:20.986799 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:51:21.708660 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:21.708627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:21.709136 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:21.708760 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:22.375681 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:22.375728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.375874 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.375898 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.375932 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:26.375912424 +0000 UTC m=+9.216914880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:22.376300 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.375961 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:26.375943149 +0000 UTC m=+9.216945607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:22.477784 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:22.477091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:22.477784 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.477367 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:22.477784 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.477390 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:22.477784 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.477403 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:22.477784 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.477459 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:26.477441004 +0000 UTC m=+9.318443469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:22.707596 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:22.707517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:22.707753 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.707655 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:22.708255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:22.708050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:22.708255 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:22.708156 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:23.711340 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:23.710849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:23.711340 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:23.711002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:24.707483 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:24.707361 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:24.707483 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:24.707406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:24.707733 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:24.707493 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:24.707793 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:24.707763 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:25.707626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:25.707129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:25.707626 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:25.707252 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:26.411747 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:26.411706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:26.411964 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:26.411761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:26.411964 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.411927 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:26.411964 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.411945 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:26.412126 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.411990 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:34.411970063 +0000 UTC m=+17.252972517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:26.412126 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.412009 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:34.411999429 +0000 UTC m=+17.253001884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:26.512762 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:26.512727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:26.512947 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.512927 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:26.513013 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.512951 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:26.513013 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.512963 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:26.513123 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.513086 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:34.513066033 +0000 UTC m=+17.354068498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:26.707644 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:26.707568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:26.708108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:26.707589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:26.708108 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.707697 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:26.708108 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:26.707788 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:27.708538 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:27.708502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:27.709022 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:27.708627 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:28.706780 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:28.706745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:28.706974 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:28.706745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:28.706974 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:28.706887 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:28.707096 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:28.706997 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:29.709978 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:29.709953 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:29.710442 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:29.710055 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:30.706627 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:30.706591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:30.706806 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:30.706605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:30.706806 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:30.706728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:30.706961 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:30.706805 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:31.707317 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:31.707249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:31.707776 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:31.707386 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:32.707438 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:32.707361 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:32.707847 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:32.707370 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:32.707847 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:32.707472 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:32.707847 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:32.707587 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:33.707348 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:33.707320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:33.707494 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:33.707416 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:34.470709 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:34.470679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:34.470709 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:34.470713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:34.470975 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.470810 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:34.470975 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.470812 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:34.470975 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.470879 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:50.470851452 +0000 UTC m=+33.311853907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:34.470975 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.470914 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:51:50.470899404 +0000 UTC m=+33.311901857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:34.571107 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:34.571082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:34.571229 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.571174 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:34.571229 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.571189 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:34.571229 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.571197 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:34.571344 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.571232 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:50.57122282 +0000 UTC m=+33.412225271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:34.707192 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:34.707161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:34.707325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:34.707172 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:34.707325 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.707286 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:34.707413 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:34.707339 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:35.706814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:35.706778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:35.707283 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:35.706915 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:36.706947 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:36.706913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:36.707415 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:36.706913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:36.707415 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:36.707035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:36.707415 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:36.707149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:37.708032 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:37.707995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:37.708458 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:37.708117 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:38.707560 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:38.707527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:38.707720 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:38.707527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:38.707720 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:38.707628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:38.707720 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:38.707702 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:39.708233 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:39.706982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:39.708233 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:39.707293 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:40.035625 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.035350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s8jgg" event={"ID":"0e347eae-c86f-4c0c-a8d5-d191b34e1048","Type":"ContainerStarted","Data":"3f34f0ea588aec0ce268bf7c72c7f16c753787cdd47ca2682ce65f2c724da9ce"} Jul 06 19:51:40.036816 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.036790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vflbj" event={"ID":"c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7","Type":"ContainerStarted","Data":"5c45d5eb7c94ad9fb9bc7077b70767416c30c721db45ac9252ca75363376896d"} Jul 06 19:51:40.038135 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.038111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" event={"ID":"813e38510b3ec379e921946926d00ffe","Type":"ContainerStarted","Data":"72beeefdfb4c90eedcd8b26b4dab4fdd494b507b6d756b054904b80fdda674a2"} Jul 06 19:51:40.039820 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.039791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" event={"ID":"933557bc-75c1-414b-841b-df6c6b75c32e","Type":"ContainerStarted","Data":"f86e2c0748feb3006f8e36b1918a9f47943d64d26ca10b8f7503667f9023a87c"} Jul 06 19:51:40.041264 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.041241 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="3903260e19b46619d23909139ae77b6fc8735d54a7b19f63d55e95a9d663ffa1" exitCode=0 Jul 06 19:51:40.041349 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.041308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"3903260e19b46619d23909139ae77b6fc8735d54a7b19f63d55e95a9d663ffa1"} Jul 06 19:51:40.042906 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.042887 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f44b43a3e849fcf8ea37cb5848fb430" containerID="189b8c3e10269afe07bff913804e03eb1d142632e4c7adbe89aefd7665788007" exitCode=0 Jul 06 19:51:40.042989 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.042951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerDied","Data":"189b8c3e10269afe07bff913804e03eb1d142632e4c7adbe89aefd7665788007"} Jul 06 19:51:40.046429 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.046407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrcnv" event={"ID":"eb4207db-8056-491e-9d95-87a6b95e9dd0","Type":"ContainerStarted","Data":"216fd1e458f99909e7da8ca70bc9c07a05c5bf87b18676fd7db79251fb920380"} Jul 06 19:51:40.048517 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.048499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5q9w2" event={"ID":"603d4e89-5bc7-4145-8fef-85bf2976fc6b","Type":"ContainerStarted","Data":"7340a1c98d70579ad846594c963e3aed6e8767f9dea4660b6ba1b0e2ffbaeca5"} Jul 06 19:51:40.049896 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.049878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" event={"ID":"aaf34c4f-6dc4-4a61-ab2a-9af3f60b78cb","Type":"ContainerStarted","Data":"a4dda2aae2a25b59f12653f39ad2ec9688d34c529300598d0b2892d0c51b4615"} Jul 06 19:51:40.052376 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052361 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:51:40.052716 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052696 2564 generic.go:356] "Generic (PLEG): container finished" podID="f57894ae-5abb-42ec-b9d1-5e4cd912760d" containerID="52d3782a47d0bde09f8beb8b1c508ebd0b46eed7080a32e7ce24e767dbe53cf8" exitCode=1 Jul 06 19:51:40.052813 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"a9069615b130f206c0a956973cb5a4dda7b4d1d45878de6738c0479c3efbb0a8"} Jul 06 19:51:40.052813 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"67b2106905da15ed142dcc34035813f980af04de944f731c6283cb9a3eda120e"} Jul 06 19:51:40.052813 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"819a779d29daff761f2bdcdf09237360adaeb203d8d9fdd6a7acbc21ea869e85"} Jul 06 19:51:40.052983 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"71bfac797b6a4ca446bb08b4d1f7947279b3b4e34be63e815c846ed439b070e5"} Jul 06 19:51:40.052983 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052852 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerDied","Data":"52d3782a47d0bde09f8beb8b1c508ebd0b46eed7080a32e7ce24e767dbe53cf8"} Jul 06 19:51:40.052983 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.052901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"5caba8e08db366efd974350ebe0979249ba4691df53f6435bce87a353a7a1ee3"} Jul 06 19:51:40.053124 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.053038 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s8jgg" podStartSLOduration=3.006749539 podStartE2EDuration="23.053021077s" totalImagesPullingTime="20.046271538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.051719915 +0000 UTC m=+22.892722387" watchObservedRunningTime="2026-07-06 19:51:40.053021077 +0000 UTC m=+22.894023550" Jul 06 19:51:40.108722 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.108669 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5q9w2" podStartSLOduration=3.183300407 podStartE2EDuration="23.10865078s" totalImagesPullingTime="19.925350373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.108183791 +0000 UTC m=+22.949186286" watchObservedRunningTime="2026-07-06 19:51:40.10865078 +0000 UTC m=+22.949653257" Jul 06 19:51:40.132404 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.132363 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rrcnv" podStartSLOduration=2.8916126220000002 podStartE2EDuration="23.132350131s" totalImagesPullingTime="20.240737509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.131415894 +0000 UTC m=+22.972418386" watchObservedRunningTime="2026-07-06 19:51:40.132350131 +0000 UTC m=+22.973352631" Jul 06 19:51:40.154164 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.154125 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-110.ec2.internal" podStartSLOduration=22.154111962 podStartE2EDuration="22.154111962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.153703187 +0000 UTC m=+22.994705660" watchObservedRunningTime="2026-07-06 19:51:40.154111962 +0000 UTC m=+22.995114434" Jul 06 19:51:40.174820 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.174777 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vflbj" podStartSLOduration=3.092888098 podStartE2EDuration="23.174763681s" totalImagesPullingTime="20.081875583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.174719904 +0000 UTC m=+23.015722377" watchObservedRunningTime="2026-07-06 19:51:40.174763681 +0000 UTC m=+23.015766154" Jul 06 19:51:40.206973 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.206929 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6dsx6" podStartSLOduration=3.257379346 podStartE2EDuration="23.206913649s" totalImagesPullingTime="19.949534303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:40.206377208 +0000 UTC m=+23.047379681" watchObservedRunningTime="2026-07-06 19:51:40.206913649 +0000 UTC m=+23.047916121" Jul 06 19:51:40.706827 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.706795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:40.707096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:40.706801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:40.707096 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:40.706923 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:40.707096 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:40.706982 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:41.056689 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.056651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6x5z" event={"ID":"b803583c-da0c-4429-826d-0e29a5109686","Type":"ContainerStarted","Data":"ca56bd1803f826196f1fb97facd27dc35c1cf372e1a71a6126a5a23ab3407ab3"} Jul 06 19:51:41.059958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.059929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" event={"ID":"7f44b43a3e849fcf8ea37cb5848fb430","Type":"ContainerStarted","Data":"ecbc523eb7c0f674e1349cc6902997b8427d7b8ba75bcd955e956f9a36a86134"} Jul 06 19:51:41.074410 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.074370 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j6x5z" podStartSLOduration=4.10489661 podStartE2EDuration="24.074356419s" totalImagesPullingTime="19.969459809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:41.073887896 +0000 UTC m=+23.914890390" watchObservedRunningTime="2026-07-06 19:51:41.074356419 +0000 UTC m=+23.915358892" Jul 06 19:51:41.130665 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.130641 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:51:41.618698 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.618582 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:51:41.130660176Z","UUID":"4710dc85-4add-4761-bac4-a8ebcc23b060","Handler":null,"Name":"","Endpoint":""} Jul 06 19:51:41.620964 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.620939 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:51:41.621092 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.620988 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:51:41.707363 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:41.707339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:41.707555 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:41.707433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:42.063647 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:42.063617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" event={"ID":"933557bc-75c1-414b-841b-df6c6b75c32e","Type":"ContainerStarted","Data":"a5c638d7aa60c48029ff8e87fa762029a9eea45c2183b231515c490d2e16beb7"} Jul 06 19:51:42.706958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:42.706926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:42.707139 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:42.706930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:42.707139 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:42.707038 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:42.707139 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:42.707133 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:43.069481 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.069403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:51:43.069931 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.069905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"74aa828b141e5b01fe0595f75ce7f33e50b17aba21da4e050fb43491e283e416"} Jul 06 19:51:43.072310 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.072275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" event={"ID":"933557bc-75c1-414b-841b-df6c6b75c32e","Type":"ContainerStarted","Data":"5ac1c27a8958327d84ee3329613387cbc742406c2e7b0e79cbb61840ccdfa87f"} Jul 06 19:51:43.089597 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.089550 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-110.ec2.internal" podStartSLOduration=25.089533627 podStartE2EDuration="25.089533627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:41.095625528 +0000 UTC m=+23.936627995" watchObservedRunningTime="2026-07-06 19:51:43.089533627 +0000 UTC m=+25.930536101" Jul 06 19:51:43.089740 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.089705 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cx2db" podStartSLOduration=3.817678311 podStartE2EDuration="26.089699507s" totalImagesPullingTime="22.272021196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:43.088962076 +0000 UTC m=+25.929964572" watchObservedRunningTime="2026-07-06 19:51:43.089699507 +0000 UTC m=+25.930701980" Jul 06 19:51:43.707319 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.707283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:43.707493 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:43.707417 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:43.918463 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.918430 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:43.919063 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:43.919045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:44.074813 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:44.074732 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:44.075266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:44.075196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vflbj" Jul 06 19:51:44.707118 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:44.707088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:44.707305 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:44.707209 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:44.707305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:44.707264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:44.707418 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:44.707399 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:45.080503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.080341 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:51:45.081181 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.080872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"754ddf1bf028f6572ecea423828dc3e16237b5bf29dac6bbd46c2ecffb930b0c"} Jul 06 19:51:45.081248 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.081171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:45.081318 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.081297 2564 scope.go:117] "RemoveContainer" containerID="52d3782a47d0bde09f8beb8b1c508ebd0b46eed7080a32e7ce24e767dbe53cf8" Jul 06 19:51:45.081369 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.081333 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:45.081417 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.081398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:45.097726 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.097707 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:45.099223 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.099125 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:51:45.706962 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:45.706926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:45.707150 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:45.707048 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:46.086941 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.086877 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:51:46.087523 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.087272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" event={"ID":"f57894ae-5abb-42ec-b9d1-5e4cd912760d","Type":"ContainerStarted","Data":"06359065c0e2063d7f5b62adfcab165b107c7e462cc3a760f599e7c57d2c6b0e"} Jul 06 19:51:46.089031 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.089008 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="d0205303c53a3bdd1c308714eaee72752dc1c6afa956a5a1a36b6bf95517c454" exitCode=0 Jul 06 19:51:46.089130 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.089090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"d0205303c53a3bdd1c308714eaee72752dc1c6afa956a5a1a36b6bf95517c454"} Jul 06 19:51:46.156202 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.156150 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" podStartSLOduration=9.152578675 podStartE2EDuration="29.156133268s" totalImagesPullingTime="20.003554593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:46.125263281 +0000 UTC m=+28.966265974" watchObservedRunningTime="2026-07-06 19:51:46.156133268 +0000 UTC m=+28.997135751" Jul 06 19:51:46.706890 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.706702 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:46.707036 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:46.706703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:46.707036 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:46.706962 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:46.707120 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:46.707050 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:47.351054 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.350974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cq5hm"] Jul 06 19:51:47.351443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.351118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:47.351443 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:47.351208 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:47.352970 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.352948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5w56"] Jul 06 19:51:47.353074 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.353031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:47.353121 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:47.353103 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:47.354586 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.354566 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zlngt"] Jul 06 19:51:47.354711 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:47.354657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:47.354764 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:47.354728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:48.095650 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:48.095612 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="caea67e5eb6aa6d5e2fe644f90fc3ba6543e5469e73c16e71a413074663107b3" exitCode=0 Jul 06 19:51:48.095905 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:48.095705 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"caea67e5eb6aa6d5e2fe644f90fc3ba6543e5469e73c16e71a413074663107b3"} Jul 06 19:51:48.707456 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:48.707428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:48.707898 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:48.707458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:48.707898 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:48.707604 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:48.707898 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:48.707708 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:49.707663 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:49.707641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:49.708035 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:49.707734 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:50.102122 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.102036 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="a148760b10b65699cb102b984613365c22fef98f58e66e90d690832b17e0c61f" exitCode=0 Jul 06 19:51:50.102122 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.102082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"a148760b10b65699cb102b984613365c22fef98f58e66e90d690832b17e0c61f"} Jul 06 19:51:50.495806 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.495779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:50.496000 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.495817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:50.496000 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.495963 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:50.496000 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.495995 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:50.496130 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.496036 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret podName:7e613ae5-5cda-463a-8a25-3b357a21de22 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:22.496016366 +0000 UTC m=+65.337018831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret") pod "global-pull-secret-syncer-t5w56" (UID: "7e613ae5-5cda-463a-8a25-3b357a21de22") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:51:50.496130 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.496059 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs podName:86913132-0316-44d0-a828-eaf00b45c59c nodeName:}" failed. No retries permitted until 2026-07-06 19:52:22.49604742 +0000 UTC m=+65.337049877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs") pod "network-metrics-daemon-cq5hm" (UID: "86913132-0316-44d0-a828-eaf00b45c59c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:51:50.596353 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.596323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:50.596486 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.596471 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:51:50.596539 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.596492 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:51:50.596539 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.596502 2564 projected.go:196] Error preparing data for projected volume kube-api-access-l8zq9 for pod openshift-network-diagnostics/network-check-target-zlngt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:50.596610 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.596546 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9 podName:86f3db8a-b66b-4332-9828-11a188a1d298 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:22.596529384 +0000 UTC m=+65.437531834 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l8zq9" (UniqueName: "kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9") pod "network-check-target-zlngt" (UID: "86f3db8a-b66b-4332-9828-11a188a1d298") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:51:50.707353 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.707277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:50.707497 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.707406 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5w56" podUID="7e613ae5-5cda-463a-8a25-3b357a21de22" Jul 06 19:51:50.707497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:50.707463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:50.707606 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:50.707579 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cq5hm" podUID="86913132-0316-44d0-a828-eaf00b45c59c" Jul 06 19:51:51.707268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:51.707090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:51.707682 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:51.707338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zlngt" podUID="86f3db8a-b66b-4332-9828-11a188a1d298" Jul 06 19:51:51.935443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:51.935419 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-110.ec2.internal" event="NodeReady" Jul 06 19:51:51.935605 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:51.935543 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:51:52.070879 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.070788 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rhkcb"] Jul 06 19:51:52.072611 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.072590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.076377 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.076336 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-vp5kp\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vp5kp\"" type="*v1.Secret" Jul 06 19:51:52.076377 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.076352 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"ingress-canary-rhkcb\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="2cb88d98-3693-4563-a9e2-99777a639030" pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.076521 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.076337 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"canary-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" Jul 06 19:51:52.076721 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.076690 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:51:52.077173 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.077147 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress-canary\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:51:52.077627 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.077605 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xxdd8"] Jul 06 19:51:52.079428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.079389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.085452 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.085434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:51:52.085556 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.085466 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:51:52.089632 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.089615 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:51:52.090574 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.090558 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:51:52.090670 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.090601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-72blj\"" Jul 06 19:51:52.110410 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.110038 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhkcb"] Jul 06 19:51:52.110921 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.110897 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xxdd8"] Jul 06 19:51:52.111598 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.111577 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r9hcp"] Jul 06 19:51:52.114142 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.114123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.119533 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.119514 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:51:52.119628 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.119515 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:51:52.119628 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.119520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzh6g\"" Jul 06 19:51:52.137638 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.137616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9hcp"] Jul 06 19:51:52.144111 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.144092 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5q9w2_603d4e89-5bc7-4145-8fef-85bf2976fc6b/dns-node-resolver/0.log" Jul 06 19:51:52.209044 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.209044 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df4f4f-a5fa-4718-b59f-59b42a080274-tmp-dir\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.209220 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda2f75a-466d-42c6-b86a-de44f19e42b4-data-volume\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.209220 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.209220 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.209220 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28df4f4f-a5fa-4718-b59f-59b42a080274-config-volume\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.209416 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7l75\" (UniqueName: \"kubernetes.io/projected/28df4f4f-a5fa-4718-b59f-59b42a080274-kube-api-access-d7l75\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.209416 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l949\" (UniqueName: \"kubernetes.io/projected/2cb88d98-3693-4563-a9e2-99777a639030-kube-api-access-2l949\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.209416 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.209416 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvgj\" (UniqueName: \"kubernetes.io/projected/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-api-access-fvvgj\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.209604 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.209414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cda2f75a-466d-42c6-b86a-de44f19e42b4-crio-socket\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310296 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df4f4f-a5fa-4718-b59f-59b42a080274-tmp-dir\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda2f75a-466d-42c6-b86a-de44f19e42b4-data-volume\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28df4f4f-a5fa-4718-b59f-59b42a080274-config-volume\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7l75\" (UniqueName: \"kubernetes.io/projected/28df4f4f-a5fa-4718-b59f-59b42a080274-kube-api-access-d7l75\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.310451 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310438 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310498 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls podName:28df4f4f-a5fa-4718-b59f-59b42a080274 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:52.810479437 +0000 UTC m=+35.651481909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls") pod "dns-default-r9hcp" (UID: "28df4f4f-a5fa-4718-b59f-59b42a080274") : secret "dns-default-metrics-tls" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310557 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l949\" (UniqueName: \"kubernetes.io/projected/2cb88d98-3693-4563-a9e2-99777a639030-kube-api-access-2l949\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310614 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:52.810596002 +0000 UTC m=+35.651598455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310632 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvgj\" (UniqueName: \"kubernetes.io/projected/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-api-access-fvvgj\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.310673 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:52.810660746 +0000 UTC m=+35.651663199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : configmap "kube-rbac-proxy" not found Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310687 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28df4f4f-a5fa-4718-b59f-59b42a080274-tmp-dir\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cda2f75a-466d-42c6-b86a-de44f19e42b4-crio-socket\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.310791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310710 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda2f75a-466d-42c6-b86a-de44f19e42b4-data-volume\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.311347 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.310932 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cda2f75a-466d-42c6-b86a-de44f19e42b4-crio-socket\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.311347 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.311129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28df4f4f-a5fa-4718-b59f-59b42a080274-config-volume\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.324460 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.324404 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7l75\" (UniqueName: \"kubernetes.io/projected/28df4f4f-a5fa-4718-b59f-59b42a080274-kube-api-access-d7l75\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.324460 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.324404 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvgj\" (UniqueName: \"kubernetes.io/projected/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-api-access-fvvgj\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.707588 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.707556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:51:52.708191 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.707623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:51:52.710193 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.710169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:51:52.710381 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.710314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c45tr\"" Jul 06 19:51:52.710381 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.710172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:51:52.815301 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.815276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:52.815431 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.815323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.815513 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815448 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:51:52.815576 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815521 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls podName:28df4f4f-a5fa-4718-b59f-59b42a080274 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:53.815501809 +0000 UTC m=+36.656504263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls") pod "dns-default-r9hcp" (UID: "28df4f4f-a5fa-4718-b59f-59b42a080274") : secret "dns-default-metrics-tls" not found Jul 06 19:51:52.815576 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815526 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:51:52.815681 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.815571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:52.815681 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815588 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:53.815571039 +0000 UTC m=+36.656573497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:51:52.815681 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815621 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:51:52.815681 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:52.815657 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:53.815644496 +0000 UTC m=+36.656646952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : configmap "kube-rbac-proxy" not found Jul 06 19:51:52.860258 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.860224 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q9855"] Jul 06 19:51:52.862648 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.862628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:52.868121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.865373 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 19:51:52.868121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.865655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 19:51:52.868121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.865746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nttxx\"" Jul 06 19:51:52.868121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.865654 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 19:51:52.868121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.865655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 19:51:52.874183 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.874151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q9855"] Jul 06 19:51:52.970001 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:52.969927 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:51:53.017050 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.017019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-key\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.017218 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.017092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-cabundle\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.017218 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.017169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrv5d\" (UniqueName: \"kubernetes.io/projected/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-kube-api-access-vrv5d\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.117737 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.117704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-cabundle\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.117928 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.117762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrv5d\" (UniqueName: \"kubernetes.io/projected/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-kube-api-access-vrv5d\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.117928 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.117837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-key\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.118478 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.118451 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-cabundle\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.120500 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.120480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-signing-key\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.127622 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.127597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrv5d\" (UniqueName: \"kubernetes.io/projected/3e9fd780-3e70-46d1-9497-4e6a62c9ce8c-kube-api-access-vrv5d\") pod \"service-ca-758f599c5-q9855\" (UID: \"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c\") " pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.176571 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.176542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q9855" Jul 06 19:51:53.279887 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.279796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vp5kp\"" Jul 06 19:51:53.311149 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.311123 2564 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: failed to sync secret cache: timed out waiting for the condition Jul 06 19:51:53.311294 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.311189 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert podName:2cb88d98-3693-4563-a9e2-99777a639030 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:53.811170007 +0000 UTC m=+36.652172463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert") pod "ingress-canary-rhkcb" (UID: "2cb88d98-3693-4563-a9e2-99777a639030") : failed to sync secret cache: timed out waiting for the condition Jul 06 19:51:53.313185 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.313152 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:51:53.313604 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.313587 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s8jgg_0e347eae-c86f-4c0c-a8d5-d191b34e1048/node-ca/0.log" Jul 06 19:51:53.523514 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.523488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:51:53.532566 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.532502 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l949\" (UniqueName: \"kubernetes.io/projected/2cb88d98-3693-4563-a9e2-99777a639030-kube-api-access-2l949\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:53.707592 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.707556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:51:53.711465 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.711444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hp2zw\"" Jul 06 19:51:53.711836 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.711444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:51:53.711836 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.711522 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:51:53.823761 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.823688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:53.823761 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.823739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.823770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.823796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.823829 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.823911 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:55.823891899 +0000 UTC m=+38.664894362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : configmap "kube-rbac-proxy" not found Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.823921 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.823949 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:51:53.823989 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.823993 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls podName:28df4f4f-a5fa-4718-b59f-59b42a080274 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:55.823971969 +0000 UTC m=+38.664974423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls") pod "dns-default-r9hcp" (UID: "28df4f4f-a5fa-4718-b59f-59b42a080274") : secret "dns-default-metrics-tls" not found Jul 06 19:51:53.824323 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:53.824014 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:55.824004423 +0000 UTC m=+38.665006877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:51:53.826483 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.826460 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cb88d98-3693-4563-a9e2-99777a639030-cert\") pod \"ingress-canary-rhkcb\" (UID: \"2cb88d98-3693-4563-a9e2-99777a639030\") " pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:53.885327 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:53.885301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rhkcb" Jul 06 19:51:54.195477 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:54.195454 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q9855"] Jul 06 19:51:54.196186 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:54.196164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rhkcb"] Jul 06 19:51:54.201198 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:54.201169 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9fd780_3e70_46d1_9497_4e6a62c9ce8c.slice/crio-76abfdc6dcf8cff1e8b8b57a42ad586b8b7516db69d8b2d39d13d4afe629ed4c WatchSource:0}: Error finding container 76abfdc6dcf8cff1e8b8b57a42ad586b8b7516db69d8b2d39d13d4afe629ed4c: Status 404 returned error can't find the container with id 76abfdc6dcf8cff1e8b8b57a42ad586b8b7516db69d8b2d39d13d4afe629ed4c Jul 06 19:51:54.201809 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:51:54.201774 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb88d98_3693_4563_a9e2_99777a639030.slice/crio-8389643c1db5dc820cf64ac45d384de8d943c2948488bdda0acbec0855e38199 WatchSource:0}: Error finding container 8389643c1db5dc820cf64ac45d384de8d943c2948488bdda0acbec0855e38199: Status 404 returned error can't find the container with id 8389643c1db5dc820cf64ac45d384de8d943c2948488bdda0acbec0855e38199 Jul 06 19:51:55.122737 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.122705 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="cb9b420c120901600cb0ad3d095f969b62d207185c2788c9c9b751088cbe61e4" exitCode=0 Jul 06 19:51:55.123475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.122797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"cb9b420c120901600cb0ad3d095f969b62d207185c2788c9c9b751088cbe61e4"} Jul 06 19:51:55.125143 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.125059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhkcb" event={"ID":"2cb88d98-3693-4563-a9e2-99777a639030","Type":"ContainerStarted","Data":"8389643c1db5dc820cf64ac45d384de8d943c2948488bdda0acbec0855e38199"} Jul 06 19:51:55.126680 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.126649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q9855" event={"ID":"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c","Type":"ContainerStarted","Data":"76abfdc6dcf8cff1e8b8b57a42ad586b8b7516db69d8b2d39d13d4afe629ed4c"} Jul 06 19:51:55.839867 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.839824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:55.840048 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.839906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:55.840048 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840017 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840051 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840076 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:55.840021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840100 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls podName:28df4f4f-a5fa-4718-b59f-59b42a080274 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:59.840077159 +0000 UTC m=+42.681079615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls") pod "dns-default-r9hcp" (UID: "28df4f4f-a5fa-4718-b59f-59b42a080274") : secret "dns-default-metrics-tls" not found Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840128 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:59.840117197 +0000 UTC m=+42.681119652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:51:55.840158 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:55.840146 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:51:59.840137544 +0000 UTC m=+42.681139997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : configmap "kube-rbac-proxy" not found Jul 06 19:51:56.133351 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:56.133317 2564 generic.go:356] "Generic (PLEG): container finished" podID="222059e5-18ec-46b0-8613-d310fc61925b" containerID="d0913a8f2cecf3e61c84664aff11ab9899562db7188d957a89d7288268981495" exitCode=0 Jul 06 19:51:56.133788 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:56.133371 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerDied","Data":"d0913a8f2cecf3e61c84664aff11ab9899562db7188d957a89d7288268981495"} Jul 06 19:51:58.144405 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.144222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" event={"ID":"222059e5-18ec-46b0-8613-d310fc61925b","Type":"ContainerStarted","Data":"222296a570481b2cf57d1fe62145f5bc340220c5560f0ff0b952208e96367874"} Jul 06 19:51:58.145647 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.145617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rhkcb" event={"ID":"2cb88d98-3693-4563-a9e2-99777a639030","Type":"ContainerStarted","Data":"0fb8aec5ab4ccebbf968741aba81f214757880a268689fae9c33f528af7ab6ea"} Jul 06 19:51:58.147044 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.147021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q9855" event={"ID":"3e9fd780-3e70-46d1-9497-4e6a62c9ce8c","Type":"ContainerStarted","Data":"a6b097d925142661f79c44495b0f4f575846abaea3ef436667c99e80c5fa2841"} Jul 06 19:51:58.170569 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.170524 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5bdqd" podStartSLOduration=10.034272062 podStartE2EDuration="41.170512073s" totalImagesPullingTime="31.136240011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:58.169986192 +0000 UTC m=+41.010988682" watchObservedRunningTime="2026-07-06 19:51:58.170512073 +0000 UTC m=+41.011514543" Jul 06 19:51:58.186687 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.186638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-q9855" podStartSLOduration=3.179673114 podStartE2EDuration="6.186624178s" totalImagesPullingTime="3.006951064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:58.186488588 +0000 UTC m=+41.027491061" watchObservedRunningTime="2026-07-06 19:51:58.186624178 +0000 UTC m=+41.027626651" Jul 06 19:51:58.201133 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:58.201094 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rhkcb" podStartSLOduration=3.200255972 podStartE2EDuration="6.20108755s" totalImagesPullingTime="3.000831578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:58.200745251 +0000 UTC m=+41.041747723" watchObservedRunningTime="2026-07-06 19:51:58.20108755 +0000 UTC m=+41.042090024" Jul 06 19:51:59.872475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:59.872441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:59.872487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:51:59.872571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872586 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872634 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872639 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls podName:28df4f4f-a5fa-4718-b59f-59b42a080274 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:07.872625303 +0000 UTC m=+50.713627754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls") pod "dns-default-r9hcp" (UID: "28df4f4f-a5fa-4718-b59f-59b42a080274") : secret "dns-default-metrics-tls" not found Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872701 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:07.872689137 +0000 UTC m=+50.713691587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872641 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:51:59.872879 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:51:59.872726 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:07.872720403 +0000 UTC m=+50.713722855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : configmap "kube-rbac-proxy" not found Jul 06 19:52:07.929911 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:07.929874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:07.930499 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:07.929923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:52:07.930499 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:07.929946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:07.930499 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:52:07.930073 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:52:07.930499 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:52:07.930137 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls podName:cda2f75a-466d-42c6-b86a-de44f19e42b4 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:23.930124086 +0000 UTC m=+66.771126537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xxdd8" (UID: "cda2f75a-466d-42c6-b86a-de44f19e42b4") : secret "insights-runtime-extractor-tls" not found Jul 06 19:52:07.930499 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:07.930479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cda2f75a-466d-42c6-b86a-de44f19e42b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:07.933947 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:07.933925 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df4f4f-a5fa-4718-b59f-59b42a080274-metrics-tls\") pod \"dns-default-r9hcp\" (UID: \"28df4f4f-a5fa-4718-b59f-59b42a080274\") " pod="openshift-dns/dns-default-r9hcp" Jul 06 19:52:08.024255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:08.024231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r9hcp" Jul 06 19:52:08.157829 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:08.156906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r9hcp"] Jul 06 19:52:08.171542 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:08.171513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9hcp" event={"ID":"28df4f4f-a5fa-4718-b59f-59b42a080274","Type":"ContainerStarted","Data":"3097d2a3abf79e561657cebe2f8e37765956a09e7bf794a36057c3effaa9c808"} Jul 06 19:52:11.180815 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:11.180776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9hcp" event={"ID":"28df4f4f-a5fa-4718-b59f-59b42a080274","Type":"ContainerStarted","Data":"87498ca14997e6a727f78f9cadac4ff544f7e2aa4ce279c393b94027a53ee5fa"} Jul 06 19:52:11.181268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:11.180835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r9hcp" event={"ID":"28df4f4f-a5fa-4718-b59f-59b42a080274","Type":"ContainerStarted","Data":"5b9aa1b6deee64cd6dbc1aec4a9d42c5ce67f9a79014d70a9d6c7da371346901"} Jul 06 19:52:11.181268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:11.180914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r9hcp" Jul 06 19:52:11.199117 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:11.199071 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r9hcp" podStartSLOduration=16.960674784 podStartE2EDuration="19.199056605s" totalImagesPullingTime="2.238381821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:11.198459008 +0000 UTC m=+54.039461496" watchObservedRunningTime="2026-07-06 19:52:11.199056605 +0000 UTC m=+54.040059097" Jul 06 19:52:17.102006 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.101973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qb4xh" Jul 06 19:52:17.647509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.647479 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm"] Jul 06 19:52:17.672380 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.672357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm"] Jul 06 19:52:17.672504 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.672439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.676495 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.676472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:52:17.676495 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.676483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nzzxs\"" Jul 06 19:52:17.676662 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.676513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:52:17.696555 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.696537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81cf6450-b4ab-44ae-80b9-66eb53ff6960-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.696650 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.696581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81cf6450-b4ab-44ae-80b9-66eb53ff6960-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.761463 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.761431 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-mp9zf"] Jul 06 19:52:17.780501 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.780475 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv"] Jul 06 19:52:17.780634 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.780604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:17.782877 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.782841 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mtlfc\"" Jul 06 19:52:17.783150 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.783127 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:52:17.783340 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.783322 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:52:17.793719 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.793701 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-mp9zf"] Jul 06 19:52:17.793834 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.793731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv"] Jul 06 19:52:17.793834 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.793747 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:52:17.793834 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.793820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:17.797027 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.797003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81cf6450-b4ab-44ae-80b9-66eb53ff6960-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.797104 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.797059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81cf6450-b4ab-44ae-80b9-66eb53ff6960-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.797147 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.797101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9th\" (UniqueName: \"kubernetes.io/projected/e50a3ad7-d449-4194-aa81-e07043b8b8f7-kube-api-access-8k9th\") pod \"downloads-685bfb5f64-mp9zf\" (UID: \"e50a3ad7-d449-4194-aa81-e07043b8b8f7\") " pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:17.797572 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.797549 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81cf6450-b4ab-44ae-80b9-66eb53ff6960-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.798186 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.798163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:52:17.798290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.798209 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 19:52:17.798290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.798219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7zgq7\"" Jul 06 19:52:17.798290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.798267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:52:17.798450 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.798211 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:52:17.799591 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.799505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81cf6450-b4ab-44ae-80b9-66eb53ff6960-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-7vnrm\" (UID: \"81cf6450-b4ab-44ae-80b9-66eb53ff6960\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.815606 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.815574 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:52:17.815717 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.815691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.818394 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.818374 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:52:17.818753 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.818705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:52:17.818845 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.818760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:52:17.818927 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.818914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zs2b6\"" Jul 06 19:52:17.819158 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.819145 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:52:17.819271 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.819253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:52:17.854406 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.854388 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f5cc55879-wkplg"] Jul 06 19:52:17.876896 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.876878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.879056 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.879035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:52:17.879403 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.879379 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-29xts\"" Jul 06 19:52:17.879509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.879436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:52:17.880956 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.880922 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:52:17.884028 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.883593 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5cc55879-wkplg"] Jul 06 19:52:17.897878 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.897822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkskk\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-kube-api-access-pkskk\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.897878 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.897847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:17.897987 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.897922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/394a8437-463d-4eab-a16e-4b13247b67f8-ca-trust-extracted\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.897987 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.897955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-trusted-ca\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.897987 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.897980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898079 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898079 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-registry-certificates\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.898079 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mdf\" (UniqueName: \"kubernetes.io/projected/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-kube-api-access-g9mdf\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:17.898079 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-installation-pull-secrets\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-bound-sa-token\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-registry-tls\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9th\" (UniqueName: \"kubernetes.io/projected/e50a3ad7-d449-4194-aa81-e07043b8b8f7-kube-api-access-8k9th\") pod \"downloads-685bfb5f64-mp9zf\" (UID: \"e50a3ad7-d449-4194-aa81-e07043b8b8f7\") " pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:17.898325 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcd9\" (UniqueName: \"kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.898536 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.898409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-image-registry-private-configuration\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.905849 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.905830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:52:17.919908 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.919889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9th\" (UniqueName: \"kubernetes.io/projected/e50a3ad7-d449-4194-aa81-e07043b8b8f7-kube-api-access-8k9th\") pod \"downloads-685bfb5f64-mp9zf\" (UID: \"e50a3ad7-d449-4194-aa81-e07043b8b8f7\") " pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:17.981291 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.981271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" Jul 06 19:52:17.998693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.998781 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.998781 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.998781 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-installation-pull-secrets\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.998971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-bound-sa-token\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.998971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-registry-tls\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.998971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcd9\" (UniqueName: \"kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.999118 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.998994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-image-registry-private-configuration\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.999118 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkskk\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-kube-api-access-pkskk\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.999118 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:17.999278 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/394a8437-463d-4eab-a16e-4b13247b67f8-ca-trust-extracted\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.999278 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-trusted-ca\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.999278 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.999278 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:17.999497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-registry-certificates\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:17.999497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mdf\" (UniqueName: \"kubernetes.io/projected/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-kube-api-access-g9mdf\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:17.999497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.000033 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:17.999931 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/394a8437-463d-4eab-a16e-4b13247b67f8-ca-trust-extracted\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.001306 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.001198 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.002035 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.001573 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-trusted-ca\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.002035 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.001591 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.002035 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.001673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-installation-pull-secrets\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.002035 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.001967 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.002548 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.002509 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/394a8437-463d-4eab-a16e-4b13247b67f8-registry-certificates\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.002963 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.002945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.003123 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.003101 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:18.004099 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.004081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-registry-tls\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.004099 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.004089 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/394a8437-463d-4eab-a16e-4b13247b67f8-image-registry-private-configuration\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.012110 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.012083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mdf\" (UniqueName: \"kubernetes.io/projected/dbd1b6f7-6746-4daa-ba83-40d1e1cd6930-kube-api-access-g9mdf\") pod \"managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv\" (UID: \"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:18.017128 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.017105 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-bound-sa-token\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.018678 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.018651 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcd9\" (UniqueName: \"kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9\") pod \"console-868dcf54cc-7mscx\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.020548 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.020530 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkskk\" (UniqueName: \"kubernetes.io/projected/394a8437-463d-4eab-a16e-4b13247b67f8-kube-api-access-pkskk\") pod \"image-registry-7f5cc55879-wkplg\" (UID: \"394a8437-463d-4eab-a16e-4b13247b67f8\") " pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.090324 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.090292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:18.117639 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.117619 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" Jul 06 19:52:18.121992 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.121970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm"] Jul 06 19:52:18.124977 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.124938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:18.125196 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:18.125176 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cf6450_b4ab_44ae_80b9_66eb53ff6960.slice/crio-f91063f6d87d6fcb36ce3ec22451a5e37f695785b214f46b615eb988f0a8f450 WatchSource:0}: Error finding container f91063f6d87d6fcb36ce3ec22451a5e37f695785b214f46b615eb988f0a8f450: Status 404 returned error can't find the container with id f91063f6d87d6fcb36ce3ec22451a5e37f695785b214f46b615eb988f0a8f450 Jul 06 19:52:18.189240 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.188905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:18.200487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.200441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" event={"ID":"81cf6450-b4ab-44ae-80b9-66eb53ff6960","Type":"ContainerStarted","Data":"f91063f6d87d6fcb36ce3ec22451a5e37f695785b214f46b615eb988f0a8f450"} Jul 06 19:52:18.254531 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.254414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-mp9zf"] Jul 06 19:52:18.259541 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:18.259515 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50a3ad7_d449_4194_aa81_e07043b8b8f7.slice/crio-084d55a5c022596387dfd9401ffc317acafb44e4e123ccc4d12e5d314ddbb588 WatchSource:0}: Error finding container 084d55a5c022596387dfd9401ffc317acafb44e4e123ccc4d12e5d314ddbb588: Status 404 returned error can't find the container with id 084d55a5c022596387dfd9401ffc317acafb44e4e123ccc4d12e5d314ddbb588 Jul 06 19:52:18.304005 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.303972 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv"] Jul 06 19:52:18.307192 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:18.307165 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbd1b6f7_6746_4daa_ba83_40d1e1cd6930.slice/crio-91ed5e848e9c2c0247afa833e4b792d1bf7a64f155375cef62f20631172d206a WatchSource:0}: Error finding container 91ed5e848e9c2c0247afa833e4b792d1bf7a64f155375cef62f20631172d206a: Status 404 returned error can't find the container with id 91ed5e848e9c2c0247afa833e4b792d1bf7a64f155375cef62f20631172d206a Jul 06 19:52:18.324406 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.324383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:52:18.327447 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:18.327424 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f90fd5_2b7b_4781_9e5b_1955fa9c8826.slice/crio-392f024e0cabfad569e3fc5381abc21d168e7a503b84aa4a1a797eb6b1de3df9 WatchSource:0}: Error finding container 392f024e0cabfad569e3fc5381abc21d168e7a503b84aa4a1a797eb6b1de3df9: Status 404 returned error can't find the container with id 392f024e0cabfad569e3fc5381abc21d168e7a503b84aa4a1a797eb6b1de3df9 Jul 06 19:52:18.367722 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:18.367696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5cc55879-wkplg"] Jul 06 19:52:18.370393 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:18.370363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394a8437_463d_4eab_a16e_4b13247b67f8.slice/crio-0abdd407ff67fd92778b3f4067710973c29facc38a2eed8f1319548c6c51007b WatchSource:0}: Error finding container 0abdd407ff67fd92778b3f4067710973c29facc38a2eed8f1319548c6c51007b: Status 404 returned error can't find the container with id 0abdd407ff67fd92778b3f4067710973c29facc38a2eed8f1319548c6c51007b Jul 06 19:52:19.209659 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.209625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" event={"ID":"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930","Type":"ContainerStarted","Data":"91ed5e848e9c2c0247afa833e4b792d1bf7a64f155375cef62f20631172d206a"} Jul 06 19:52:19.212575 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.212544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" event={"ID":"394a8437-463d-4eab-a16e-4b13247b67f8","Type":"ContainerStarted","Data":"e16540184d2d534eebd0babc22d88a77ecddf93f7d4046649ff901423a638e8f"} Jul 06 19:52:19.212706 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.212595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" event={"ID":"394a8437-463d-4eab-a16e-4b13247b67f8","Type":"ContainerStarted","Data":"0abdd407ff67fd92778b3f4067710973c29facc38a2eed8f1319548c6c51007b"} Jul 06 19:52:19.212706 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.212623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:19.214738 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.214712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868dcf54cc-7mscx" event={"ID":"97f90fd5-2b7b-4781-9e5b-1955fa9c8826","Type":"ContainerStarted","Data":"392f024e0cabfad569e3fc5381abc21d168e7a503b84aa4a1a797eb6b1de3df9"} Jul 06 19:52:19.216741 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:19.216715 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-mp9zf" event={"ID":"e50a3ad7-d449-4194-aa81-e07043b8b8f7","Type":"ContainerStarted","Data":"084d55a5c022596387dfd9401ffc317acafb44e4e123ccc4d12e5d314ddbb588"} Jul 06 19:52:20.224927 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:20.224885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" event={"ID":"81cf6450-b4ab-44ae-80b9-66eb53ff6960","Type":"ContainerStarted","Data":"9512b0554fee313e5cde7f2b8a97eddeb592334d41d864a37f962bd1ca08da5b"} Jul 06 19:52:20.257080 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:20.257023 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" podStartSLOduration=3.257002577 podStartE2EDuration="3.257002577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:19.237896729 +0000 UTC m=+62.078899203" watchObservedRunningTime="2026-07-06 19:52:20.257002577 +0000 UTC m=+63.098005050" Jul 06 19:52:20.257367 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:20.257328 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-7vnrm" podStartSLOduration=1.563046129 podStartE2EDuration="3.257312452s" totalImagesPullingTime="1.694266323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:20.253842456 +0000 UTC m=+63.094844952" watchObservedRunningTime="2026-07-06 19:52:20.257312452 +0000 UTC m=+63.098314927" Jul 06 19:52:21.056417 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.056384 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6"] Jul 06 19:52:21.058710 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.058684 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.060952 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.060930 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 19:52:21.061533 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.061504 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:52:21.061885 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.061770 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:52:21.061885 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.061847 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:52:21.062038 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.061996 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2qqwg\"" Jul 06 19:52:21.062100 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.062074 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 19:52:21.071537 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.071514 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6"] Jul 06 19:52:21.125816 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.125782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndbw\" (UniqueName: \"kubernetes.io/projected/760167f0-e709-4890-be0f-9c4d9ca2a2f9-kube-api-access-pndbw\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.125996 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.125916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/760167f0-e709-4890-be0f-9c4d9ca2a2f9-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.125996 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.125956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.125996 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.125990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.185010 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.184978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r9hcp" Jul 06 19:52:21.227099 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.227061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/760167f0-e709-4890-be0f-9c4d9ca2a2f9-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.227509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.227140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.227509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.227185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.227509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.227237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndbw\" (UniqueName: \"kubernetes.io/projected/760167f0-e709-4890-be0f-9c4d9ca2a2f9-kube-api-access-pndbw\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.229238 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.229184 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/760167f0-e709-4890-be0f-9c4d9ca2a2f9-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.235801 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.235753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.235961 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.235830 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/760167f0-e709-4890-be0f-9c4d9ca2a2f9-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.238987 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.238852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndbw\" (UniqueName: \"kubernetes.io/projected/760167f0-e709-4890-be0f-9c4d9ca2a2f9-kube-api-access-pndbw\") pod \"prometheus-operator-567f7b8b7c-r67v6\" (UID: \"760167f0-e709-4890-be0f-9c4d9ca2a2f9\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:21.371273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:21.371227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" Jul 06 19:52:22.537654 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.537332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:52:22.537654 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.537388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:52:22.541811 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.541785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:52:22.542007 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.541990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:52:22.551833 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.551741 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86913132-0316-44d0-a828-eaf00b45c59c-metrics-certs\") pod \"network-metrics-daemon-cq5hm\" (UID: \"86913132-0316-44d0-a828-eaf00b45c59c\") " pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:52:22.552887 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.552521 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e613ae5-5cda-463a-8a25-3b357a21de22-original-pull-secret\") pod \"global-pull-secret-syncer-t5w56\" (UID: \"7e613ae5-5cda-463a-8a25-3b357a21de22\") " pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:52:22.573101 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.572549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6"] Jul 06 19:52:22.638295 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.638260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:52:22.641239 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.641210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:52:22.651035 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.650999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:52:22.662569 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.662547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zq9\" (UniqueName: \"kubernetes.io/projected/86f3db8a-b66b-4332-9828-11a188a1d298-kube-api-access-l8zq9\") pod \"network-check-target-zlngt\" (UID: \"86f3db8a-b66b-4332-9828-11a188a1d298\") " pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:52:22.721674 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.721595 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c45tr\"" Jul 06 19:52:22.724358 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.724345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5w56" Jul 06 19:52:22.730731 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.730713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cq5hm" Jul 06 19:52:22.820671 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.820644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hp2zw\"" Jul 06 19:52:22.828255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.828227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:52:22.861807 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.861737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5w56"] Jul 06 19:52:22.865906 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:22.865879 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e613ae5_5cda_463a_8a25_3b357a21de22.slice/crio-8899ff5b803cd64e41a2d2e7f08e919c493b197b77dcbb30193ef63f5fb28cc4 WatchSource:0}: Error finding container 8899ff5b803cd64e41a2d2e7f08e919c493b197b77dcbb30193ef63f5fb28cc4: Status 404 returned error can't find the container with id 8899ff5b803cd64e41a2d2e7f08e919c493b197b77dcbb30193ef63f5fb28cc4 Jul 06 19:52:22.882491 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.882461 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cq5hm"] Jul 06 19:52:22.887078 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:22.887052 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86913132_0316_44d0_a828_eaf00b45c59c.slice/crio-9e15a897a63a97a7f80f6b8b74d8d41f3a9fda2c4af9185c2792015121a33c72 WatchSource:0}: Error finding container 9e15a897a63a97a7f80f6b8b74d8d41f3a9fda2c4af9185c2792015121a33c72: Status 404 returned error can't find the container with id 9e15a897a63a97a7f80f6b8b74d8d41f3a9fda2c4af9185c2792015121a33c72 Jul 06 19:52:22.981939 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:22.981850 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zlngt"] Jul 06 19:52:22.984950 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:22.984921 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f3db8a_b66b_4332_9828_11a188a1d298.slice/crio-cc790e8016f1b36b4bc510fdfeed50ab555ed99b8ac8fc6fdee770e67919ad29 WatchSource:0}: Error finding container cc790e8016f1b36b4bc510fdfeed50ab555ed99b8ac8fc6fdee770e67919ad29: Status 404 returned error can't find the container with id cc790e8016f1b36b4bc510fdfeed50ab555ed99b8ac8fc6fdee770e67919ad29 Jul 06 19:52:23.239671 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.239580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" event={"ID":"dbd1b6f7-6746-4daa-ba83-40d1e1cd6930","Type":"ContainerStarted","Data":"4a39e1b20e7ba340744e36c68453fb38cc9ee34f3ffe112028add4d36329b567"} Jul 06 19:52:23.241229 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.241195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cq5hm" event={"ID":"86913132-0316-44d0-a828-eaf00b45c59c","Type":"ContainerStarted","Data":"9e15a897a63a97a7f80f6b8b74d8d41f3a9fda2c4af9185c2792015121a33c72"} Jul 06 19:52:23.243008 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.242964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5w56" event={"ID":"7e613ae5-5cda-463a-8a25-3b357a21de22","Type":"ContainerStarted","Data":"8899ff5b803cd64e41a2d2e7f08e919c493b197b77dcbb30193ef63f5fb28cc4"} Jul 06 19:52:23.245587 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.245526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868dcf54cc-7mscx" event={"ID":"97f90fd5-2b7b-4781-9e5b-1955fa9c8826","Type":"ContainerStarted","Data":"42bdffc93f94ea9b1f502427aea6864efdf29d4fd888125f16a3eefb0502895d"} Jul 06 19:52:23.247218 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.247171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zlngt" event={"ID":"86f3db8a-b66b-4332-9828-11a188a1d298","Type":"ContainerStarted","Data":"cc790e8016f1b36b4bc510fdfeed50ab555ed99b8ac8fc6fdee770e67919ad29"} Jul 06 19:52:23.248723 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.248696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" event={"ID":"760167f0-e709-4890-be0f-9c4d9ca2a2f9","Type":"ContainerStarted","Data":"eee24da293310e9322295761ffa8d3cb93fe49dd253754f7258b10b9b370a4e8"} Jul 06 19:52:23.257342 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.257279 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5f7dcff6-bnbrv" podStartSLOduration=2.131806027 podStartE2EDuration="6.257261327s" totalImagesPullingTime="4.1254553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:23.255645051 +0000 UTC m=+66.096647520" watchObservedRunningTime="2026-07-06 19:52:23.257261327 +0000 UTC m=+66.098263794" Jul 06 19:52:23.277454 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.277315 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868dcf54cc-7mscx" podStartSLOduration=2.168155366 podStartE2EDuration="6.27729784s" totalImagesPullingTime="4.109142474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:23.276553135 +0000 UTC m=+66.117555608" watchObservedRunningTime="2026-07-06 19:52:23.27729784 +0000 UTC m=+66.118300292" Jul 06 19:52:23.432829 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.431593 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:52:23.434025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.433995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.442798 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.442774 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:52:23.451720 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.451660 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmn4\" (UniqueName: \"kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.544498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.544342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmn4\" (UniqueName: \"kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644827 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.644961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.645725 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.647273 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.646720 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.648087 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.648043 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.648398 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.648315 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.654784 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.654721 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.658956 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.658896 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.659153 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.659128 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmn4\" (UniqueName: \"kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4\") pod \"console-5f55f66696-4qb57\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.763421 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.763387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:23.944984 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.943873 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:52:23.948699 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.948256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:23.952254 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:23.952203 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cda2f75a-466d-42c6-b86a-de44f19e42b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxdd8\" (UID: \"cda2f75a-466d-42c6-b86a-de44f19e42b4\") " pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:24.194371 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:24.194334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-72blj\"" Jul 06 19:52:24.200894 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:24.200683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xxdd8" Jul 06 19:52:24.760393 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:24.760363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc68077_549f_45c4_ab99_9b47aa300e05.slice/crio-aa7caa25f2735c2726d5593fbae1cd6c7bcd20aee1573a6196f94bdb9d4bb550 WatchSource:0}: Error finding container aa7caa25f2735c2726d5593fbae1cd6c7bcd20aee1573a6196f94bdb9d4bb550: Status 404 returned error can't find the container with id aa7caa25f2735c2726d5593fbae1cd6c7bcd20aee1573a6196f94bdb9d4bb550 Jul 06 19:52:25.263635 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:25.263544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f55f66696-4qb57" event={"ID":"3bc68077-549f-45c4-ab99-9b47aa300e05","Type":"ContainerStarted","Data":"aa7caa25f2735c2726d5593fbae1cd6c7bcd20aee1573a6196f94bdb9d4bb550"} Jul 06 19:52:25.313379 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:25.313295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xxdd8"] Jul 06 19:52:25.316232 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:25.316179 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda2f75a_466d_42c6_b86a_de44f19e42b4.slice/crio-04c7366821e2aa424cafedfe0f82274013bd73e1a9ba6e5e49fa20a980753426 WatchSource:0}: Error finding container 04c7366821e2aa424cafedfe0f82274013bd73e1a9ba6e5e49fa20a980753426: Status 404 returned error can't find the container with id 04c7366821e2aa424cafedfe0f82274013bd73e1a9ba6e5e49fa20a980753426 Jul 06 19:52:26.276400 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.276360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cq5hm" event={"ID":"86913132-0316-44d0-a828-eaf00b45c59c","Type":"ContainerStarted","Data":"368cdbac38e2452ea8ed58fe7ee2cba4d7bd758d3b1f60fb2d7ec1b2b0df84ca"} Jul 06 19:52:26.276919 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.276430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cq5hm" event={"ID":"86913132-0316-44d0-a828-eaf00b45c59c","Type":"ContainerStarted","Data":"a69e2c6d325d98c25663ad67145511b8406e24f89be825e2d56d4ac6a0778f84"} Jul 06 19:52:26.284241 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.284159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxdd8" event={"ID":"cda2f75a-466d-42c6-b86a-de44f19e42b4","Type":"ContainerStarted","Data":"04c7366821e2aa424cafedfe0f82274013bd73e1a9ba6e5e49fa20a980753426"} Jul 06 19:52:26.287697 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.287107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" event={"ID":"760167f0-e709-4890-be0f-9c4d9ca2a2f9","Type":"ContainerStarted","Data":"dddbeb1db8f4e3ff0906d0d1dc0fb7b3578b1c3a23f6f53c465df50be7070564"} Jul 06 19:52:26.287697 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.287158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" event={"ID":"760167f0-e709-4890-be0f-9c4d9ca2a2f9","Type":"ContainerStarted","Data":"7f4d31f6e8ebb489a68275d26c3963b926fcd1235b23d4fe07e354211385c6d3"} Jul 06 19:52:26.290880 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.290485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f55f66696-4qb57" event={"ID":"3bc68077-549f-45c4-ab99-9b47aa300e05","Type":"ContainerStarted","Data":"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605"} Jul 06 19:52:26.295997 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.295619 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cq5hm" podStartSLOduration=67.037318716 podStartE2EDuration="1m9.295602221s" totalImagesPullingTime="2.258283505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:26.293552151 +0000 UTC m=+69.134554649" watchObservedRunningTime="2026-07-06 19:52:26.295602221 +0000 UTC m=+69.136604695" Jul 06 19:52:26.313193 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.313145 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f55f66696-4qb57" podStartSLOduration=3.3131273070000002 podStartE2EDuration="3.313127307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:26.311891211 +0000 UTC m=+69.152893713" watchObservedRunningTime="2026-07-06 19:52:26.313127307 +0000 UTC m=+69.154129780" Jul 06 19:52:26.334761 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:26.334154 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r67v6" podStartSLOduration=2.767201481 podStartE2EDuration="5.334136828s" totalImagesPullingTime="2.566935347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:26.332911023 +0000 UTC m=+69.173913496" watchObservedRunningTime="2026-07-06 19:52:26.334136828 +0000 UTC m=+69.175139285" Jul 06 19:52:28.125975 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.125890 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:28.126443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.125993 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:28.133520 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.133480 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:28.304039 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.304013 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:28.518530 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.518447 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vlw2j"] Jul 06 19:52:28.522360 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.521404 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.526605 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.526578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4dfxs\"" Jul 06 19:52:28.527111 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.526788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:52:28.527111 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.526982 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:52:28.528663 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.528546 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-root\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-metrics-client-ca\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-textfile\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-wtmp\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-sys\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.589960 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.589830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jvx\" (UniqueName: \"kubernetes.io/projected/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-kube-api-access-57jvx\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-root\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-metrics-client-ca\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-textfile\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-wtmp\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690781 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-root\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690843 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-sys\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:52:28.690956 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:52:28.691024 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls podName:1bdc1a9d-0481-4dfa-9989-0aac73e2c444 nodeName:}" failed. No retries permitted until 2026-07-06 19:52:29.191004154 +0000 UTC m=+72.032006619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls") pod "node-exporter-vlw2j" (UID: "1bdc1a9d-0481-4dfa-9989-0aac73e2c444") : secret "node-exporter-tls" not found Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-textfile\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.690789 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-sys\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-wtmp\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.691882 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.692696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jvx\" (UniqueName: \"kubernetes.io/projected/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-kube-api-access-57jvx\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.692696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691751 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-metrics-client-ca\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.692696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.691825 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.695301 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.695260 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:28.705952 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:28.705929 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jvx\" (UniqueName: \"kubernetes.io/projected/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-kube-api-access-57jvx\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:29.196195 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.196162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:29.199559 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.199533 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bdc1a9d-0481-4dfa-9989-0aac73e2c444-node-exporter-tls\") pod \"node-exporter-vlw2j\" (UID: \"1bdc1a9d-0481-4dfa-9989-0aac73e2c444\") " pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:29.440727 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.440638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vlw2j" Jul 06 19:52:29.588726 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.588653 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:52:29.593218 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.593192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.596198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.595919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:52:29.596198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.595936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:52:29.596198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.595936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:52:29.596198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.596062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:52:29.596487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.596459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:52:29.596951 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.596697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:52:29.596951 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.596710 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l2chf\"" Jul 06 19:52:29.597219 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.596963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:52:29.597219 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.597058 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:52:29.597219 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.597139 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:52:29.609070 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.609028 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:52:29.699744 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.699943 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.699943 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.699943 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.699943 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.699982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsd4\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700415 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700415 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.700415 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.700256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.800888 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.800795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsd4\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801089 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.800850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801262 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801262 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801262 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801262 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801262 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801255 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801339 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.801623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.801422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.802984 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.802959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.803885 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.803847 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.805445 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.805419 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.806443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.806142 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.806938 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.806596 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.806938 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.806844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.806938 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.806893 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.807845 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.807791 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.809720 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.809693 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.810028 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.809971 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.810604 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.810564 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.813579 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.813531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsd4\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4\") pod \"alertmanager-main-0\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:29.905607 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:29.905577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:52:32.971050 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.971013 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85f5fbfd6d-48sr6"] Jul 06 19:52:32.973950 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.973922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:32.976905 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.976884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:52:32.977281 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.977259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:52:32.977402 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.977312 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:52:32.977402 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.977334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vhwv9\"" Jul 06 19:52:32.977402 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.977399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:52:32.977723 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.977648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-256scl3c2vfku\"" Jul 06 19:52:32.985950 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:32.985900 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85f5fbfd6d-48sr6"] Jul 06 19:52:33.032292 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-client-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032441 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-tls\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032441 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/17b08d44-62c2-43e2-9536-b378b5c4c7e6-audit-log\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032441 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-metrics-server-audit-profiles\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032441 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-client-certs\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2mc\" (UniqueName: \"kubernetes.io/projected/17b08d44-62c2-43e2-9536-b378b5c4c7e6-kube-api-access-mk2mc\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.032616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.032483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133312 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2mc\" (UniqueName: \"kubernetes.io/projected/17b08d44-62c2-43e2-9536-b378b5c4c7e6-kube-api-access-mk2mc\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-client-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-tls\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/17b08d44-62c2-43e2-9536-b378b5c4c7e6-audit-log\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-metrics-server-audit-profiles\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-client-certs\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.133986 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.133932 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/17b08d44-62c2-43e2-9536-b378b5c4c7e6-audit-log\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.134329 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.134229 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.134658 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.134637 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/17b08d44-62c2-43e2-9536-b378b5c4c7e6-metrics-server-audit-profiles\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.136360 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.136328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-tls\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.136466 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.136417 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-client-ca-bundle\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.136527 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.136492 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/17b08d44-62c2-43e2-9536-b378b5c4c7e6-secret-metrics-server-client-certs\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.141750 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.141730 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2mc\" (UniqueName: \"kubernetes.io/projected/17b08d44-62c2-43e2-9536-b378b5c4c7e6-kube-api-access-mk2mc\") pod \"metrics-server-85f5fbfd6d-48sr6\" (UID: \"17b08d44-62c2-43e2-9536-b378b5c4c7e6\") " pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.285823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.285750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:33.738589 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.738551 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5845c9cd78-t4cn8"] Jul 06 19:52:33.740832 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.740810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.743099 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743073 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 19:52:33.743229 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743167 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 19:52:33.743297 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743241 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 19:52:33.743297 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t5sk2\"" Jul 06 19:52:33.743766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743747 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 19:52:33.743766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.743753 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 19:52:33.756302 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.754985 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 19:52:33.756745 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.756652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5845c9cd78-t4cn8"] Jul 06 19:52:33.763813 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.763792 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:33.763946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.763836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:33.770298 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.770280 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:33.840041 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8qx\" (UniqueName: \"kubernetes.io/projected/a53dd33d-2495-415d-8bba-c0735f0d8907-kube-api-access-wl8qx\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840156 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840230 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-serving-certs-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840230 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-federate-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-metrics-client-ca\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.840461 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.840357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941353 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-metrics-client-ca\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941353 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941565 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941565 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8qx\" (UniqueName: \"kubernetes.io/projected/a53dd33d-2495-415d-8bba-c0735f0d8907-kube-api-access-wl8qx\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941565 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941565 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-serving-certs-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941565 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.941824 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.941575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-federate-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.942384 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.942320 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-metrics-client-ca\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.942384 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.942324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-serving-certs-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.942566 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.942437 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.944437 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.944414 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-federate-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.944550 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.944483 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.944780 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.944761 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.944982 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.944959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a53dd33d-2495-415d-8bba-c0735f0d8907-telemeter-client-tls\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:33.949077 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:33.949051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8qx\" (UniqueName: \"kubernetes.io/projected/a53dd33d-2495-415d-8bba-c0735f0d8907-kube-api-access-wl8qx\") pod \"telemeter-client-5845c9cd78-t4cn8\" (UID: \"a53dd33d-2495-415d-8bba-c0735f0d8907\") " pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:34.057762 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.057681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" Jul 06 19:52:34.326741 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.326669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:52:34.380807 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.380775 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:52:34.884108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.882652 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:52:34.888402 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.888377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.890562 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.890542 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:52:34.892972 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.892943 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-frjqx\"" Jul 06 19:52:34.893319 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.893300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:52:34.894000 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.893950 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:52:34.894124 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:52:34.894218 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894203 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:52:34.894344 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894319 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:52:34.894552 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:52:34.894665 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894645 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:52:34.894734 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894675 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:52:34.894790 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:52:34.895093 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:52:34.895093 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894976 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:52:34.895093 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.894988 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5466k88mgb2kl\"" Jul 06 19:52:34.895467 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.895449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:52:34.899250 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.899233 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:52:34.914744 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.914719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:52:34.952089 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjbm\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952089 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952270 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952270 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952270 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952270 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952439 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952439 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952439 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952439 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:34.952913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:34.952788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054161 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054350 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054646 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054646 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054646 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054552 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054874 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054957 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.054957 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjbm\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055056 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055056 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.054983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055056 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055011 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055056 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055137 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055255 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055547 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055320 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.055889 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.055844 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.056266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.056244 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.059935 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.060449 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.061497 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.061607 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.061663 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.061754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.062038 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.062092 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.062493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.062310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.063250 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.062553 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.063445 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.063400 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.063445 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.063422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.064366 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.064345 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.065401 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.065374 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjbm\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm\") pod \"prometheus-k8s-0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:35.202189 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:35.202098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:39.673894 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:39.673848 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdc1a9d_0481_4dfa_9989_0aac73e2c444.slice/crio-c8f5b3c0cec1c1d5cf9ab23ffd8208cc79d0a6d59c2e58b22f94787b7708061e WatchSource:0}: Error finding container c8f5b3c0cec1c1d5cf9ab23ffd8208cc79d0a6d59c2e58b22f94787b7708061e: Status 404 returned error can't find the container with id c8f5b3c0cec1c1d5cf9ab23ffd8208cc79d0a6d59c2e58b22f94787b7708061e Jul 06 19:52:39.884474 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:39.884445 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:52:39.889723 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:39.889696 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7c64ef_65d9_4119_b0c5_4f5d097a299c.slice/crio-c9f61b4985959ac018e84c77e2e92778a66f19ad8b3f7558bd1d8ccd502bf366 WatchSource:0}: Error finding container c9f61b4985959ac018e84c77e2e92778a66f19ad8b3f7558bd1d8ccd502bf366: Status 404 returned error can't find the container with id c9f61b4985959ac018e84c77e2e92778a66f19ad8b3f7558bd1d8ccd502bf366 Jul 06 19:52:40.115420 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.115383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5845c9cd78-t4cn8"] Jul 06 19:52:40.116887 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.116848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85f5fbfd6d-48sr6"] Jul 06 19:52:40.119649 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:40.119620 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53dd33d_2495_415d_8bba_c0735f0d8907.slice/crio-0a1493daab3a4167c2bdd14146c5101f9dd1edf097e733c25b51f572f8eb1182 WatchSource:0}: Error finding container 0a1493daab3a4167c2bdd14146c5101f9dd1edf097e733c25b51f572f8eb1182: Status 404 returned error can't find the container with id 0a1493daab3a4167c2bdd14146c5101f9dd1edf097e733c25b51f572f8eb1182 Jul 06 19:52:40.120402 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:40.120354 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b08d44_62c2_43e2_9536_b378b5c4c7e6.slice/crio-26637e05c2d4454b110660a2d9630f30d85212767220a30d9dd2d5a12f6af7d1 WatchSource:0}: Error finding container 26637e05c2d4454b110660a2d9630f30d85212767220a30d9dd2d5a12f6af7d1: Status 404 returned error can't find the container with id 26637e05c2d4454b110660a2d9630f30d85212767220a30d9dd2d5a12f6af7d1 Jul 06 19:52:40.123436 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.123264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:52:40.128031 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:52:40.128009 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d4232e_ab3d_4f03_ac95_8186252efcf0.slice/crio-99c732dcb79fbd5b93d8965cf7a098bfaa0bc03232a49054463126d3a4409d1e WatchSource:0}: Error finding container 99c732dcb79fbd5b93d8965cf7a098bfaa0bc03232a49054463126d3a4409d1e: Status 404 returned error can't find the container with id 99c732dcb79fbd5b93d8965cf7a098bfaa0bc03232a49054463126d3a4409d1e Jul 06 19:52:40.229507 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.229478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f5cc55879-wkplg" Jul 06 19:52:40.349538 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.348723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"c9f61b4985959ac018e84c77e2e92778a66f19ad8b3f7558bd1d8ccd502bf366"} Jul 06 19:52:40.350581 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.350548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" event={"ID":"17b08d44-62c2-43e2-9536-b378b5c4c7e6","Type":"ContainerStarted","Data":"26637e05c2d4454b110660a2d9630f30d85212767220a30d9dd2d5a12f6af7d1"} Jul 06 19:52:40.353573 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.353507 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlw2j" event={"ID":"1bdc1a9d-0481-4dfa-9989-0aac73e2c444","Type":"ContainerStarted","Data":"c8f5b3c0cec1c1d5cf9ab23ffd8208cc79d0a6d59c2e58b22f94787b7708061e"} Jul 06 19:52:40.361117 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.360398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-mp9zf" event={"ID":"e50a3ad7-d449-4194-aa81-e07043b8b8f7","Type":"ContainerStarted","Data":"f4b199068e3f1be231778d569dce2da29b7a7521a2906ff49d99e93bfe5623ed"} Jul 06 19:52:40.361117 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.361114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:40.363521 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.363477 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zlngt" event={"ID":"86f3db8a-b66b-4332-9828-11a188a1d298","Type":"ContainerStarted","Data":"3b2f7d4b547f4012af986213e2eddd5034f4692a1cce02bf374005bad289f3bd"} Jul 06 19:52:40.363753 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.363732 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:52:40.365847 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.365785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxdd8" event={"ID":"cda2f75a-466d-42c6-b86a-de44f19e42b4","Type":"ContainerStarted","Data":"1dbdeb77459ab2b6ee26a567240aeef5a6e13c823886eed05b78ed07cba306cd"} Jul 06 19:52:40.367742 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.367710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" event={"ID":"a53dd33d-2495-415d-8bba-c0735f0d8907","Type":"ContainerStarted","Data":"0a1493daab3a4167c2bdd14146c5101f9dd1edf097e733c25b51f572f8eb1182"} Jul 06 19:52:40.369649 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.369290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5w56" event={"ID":"7e613ae5-5cda-463a-8a25-3b357a21de22","Type":"ContainerStarted","Data":"31eb819311d98e25c46bb8ccc46bd6c452c56ee1368aca5213b8e71799448296"} Jul 06 19:52:40.370896 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.370846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"99c732dcb79fbd5b93d8965cf7a098bfaa0bc03232a49054463126d3a4409d1e"} Jul 06 19:52:40.375789 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.375758 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-mp9zf" Jul 06 19:52:40.378783 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.378206 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-mp9zf" podStartSLOduration=1.958421805 podStartE2EDuration="23.378188295s" totalImagesPullingTime="21.41976649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:40.376810705 +0000 UTC m=+83.217813203" watchObservedRunningTime="2026-07-06 19:52:40.378188295 +0000 UTC m=+83.219190770" Jul 06 19:52:40.418025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.416966 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zlngt" podStartSLOduration=67.16139053 podStartE2EDuration="1m23.416945658s" totalImagesPullingTime="16.255555128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:40.415981993 +0000 UTC m=+83.256984467" watchObservedRunningTime="2026-07-06 19:52:40.416945658 +0000 UTC m=+83.257948132" Jul 06 19:52:40.418025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:40.417722 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t5w56" podStartSLOduration=67.000521717 podStartE2EDuration="1m23.417710176s" totalImagesPullingTime="16.417188459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:40.397117013 +0000 UTC m=+83.238119487" watchObservedRunningTime="2026-07-06 19:52:40.417710176 +0000 UTC m=+83.258712650" Jul 06 19:52:43.396067 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:43.395195 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa" exitCode=0 Jul 06 19:52:43.396067 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:43.395559 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa"} Jul 06 19:52:43.399479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:43.399454 2564 generic.go:356] "Generic (PLEG): container finished" podID="1bdc1a9d-0481-4dfa-9989-0aac73e2c444" containerID="bdf3ca0dd5eabeb87432206e1f58afe4d50ecd5123ee630ca91e787e3df815fc" exitCode=0 Jul 06 19:52:43.399570 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:43.399552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlw2j" event={"ID":"1bdc1a9d-0481-4dfa-9989-0aac73e2c444","Type":"ContainerDied","Data":"bdf3ca0dd5eabeb87432206e1f58afe4d50ecd5123ee630ca91e787e3df815fc"} Jul 06 19:52:43.403576 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:43.403544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxdd8" event={"ID":"cda2f75a-466d-42c6-b86a-de44f19e42b4","Type":"ContainerStarted","Data":"d1ce7bf3995e6d8be5111afc485b3e320e23c4d4cd75f172eae03d7d1e37f928"} Jul 06 19:52:45.421043 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.421001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" event={"ID":"a53dd33d-2495-415d-8bba-c0735f0d8907","Type":"ContainerStarted","Data":"29ce299eac3739c49b0d74e37b45b79672a96e48b00741a522b861e790bb1301"} Jul 06 19:52:45.421628 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.421115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" event={"ID":"a53dd33d-2495-415d-8bba-c0735f0d8907","Type":"ContainerStarted","Data":"2a09e1c7c3c8b8541e23c15ea72fed92665cc1f7eb175bf0b3039571dfad1580"} Jul 06 19:52:45.421628 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.421146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" event={"ID":"a53dd33d-2495-415d-8bba-c0735f0d8907","Type":"ContainerStarted","Data":"6b1a3ec55560702e5661501a9cb4a48f5b10d1b240781b95467712ded92b875a"} Jul 06 19:52:45.424411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.423475 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" exitCode=0 Jul 06 19:52:45.424411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.423591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} Jul 06 19:52:45.427004 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.426978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" event={"ID":"17b08d44-62c2-43e2-9536-b378b5c4c7e6","Type":"ContainerStarted","Data":"cff8a4df73fe89c13c34f719b0ece681df96fa23d6fd2ba001ce9850449947ea"} Jul 06 19:52:45.430089 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.430012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlw2j" event={"ID":"1bdc1a9d-0481-4dfa-9989-0aac73e2c444","Type":"ContainerStarted","Data":"2bd3ad6f7ff9958096b91cf3ecff2ef027e41b5f1cceb9694dcd2785631d6e09"} Jul 06 19:52:45.430089 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.430057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlw2j" event={"ID":"1bdc1a9d-0481-4dfa-9989-0aac73e2c444","Type":"ContainerStarted","Data":"e2b4afb59c3f8a81f355cca10d6c08bd02870cbc5e4eaac9cb2220ce893cd18d"} Jul 06 19:52:45.450696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.450639 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5845c9cd78-t4cn8" podStartSLOduration=8.246177502 podStartE2EDuration="12.450621504s" totalImagesPullingTime="4.204444002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:45.449967959 +0000 UTC m=+88.290970432" watchObservedRunningTime="2026-07-06 19:52:45.450621504 +0000 UTC m=+88.291623980" Jul 06 19:52:45.473178 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.473118 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vlw2j" podStartSLOduration=14.591565729 podStartE2EDuration="17.473100285s" totalImagesPullingTime="2.749364464s" totalInitContainerRuntime="132.170092ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:45.472526451 +0000 UTC m=+88.313528958" watchObservedRunningTime="2026-07-06 19:52:45.473100285 +0000 UTC m=+88.314102763" Jul 06 19:52:45.534321 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:45.534271 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" podStartSLOduration=9.325630114 podStartE2EDuration="13.534251636s" totalImagesPullingTime="4.208621522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:45.530897968 +0000 UTC m=+88.371900470" watchObservedRunningTime="2026-07-06 19:52:45.534251636 +0000 UTC m=+88.375254111" Jul 06 19:52:48.449297 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.449241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxdd8" event={"ID":"cda2f75a-466d-42c6-b86a-de44f19e42b4","Type":"ContainerStarted","Data":"04f3edcbfec11d7aedb3ff353233ed3adb1157d92ba7a1f6d2b4a29c5e6f8579"} Jul 06 19:52:48.452947 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.452923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0"} Jul 06 19:52:48.453064 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.452991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80"} Jul 06 19:52:48.453064 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.453033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c"} Jul 06 19:52:48.453173 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.453071 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc"} Jul 06 19:52:48.453173 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.453110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44"} Jul 06 19:52:48.472084 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:48.471998 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xxdd8" podStartSLOduration=34.425469526 podStartE2EDuration="56.471980198s" totalImagesPullingTime="22.046510672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:48.469260071 +0000 UTC m=+91.310262562" watchObservedRunningTime="2026-07-06 19:52:48.471980198 +0000 UTC m=+91.312982670" Jul 06 19:52:50.468368 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:50.468334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerStarted","Data":"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7"} Jul 06 19:52:50.526081 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:50.526021 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=13.375870091 podStartE2EDuration="21.52599955s" totalImagesPullingTime="8.121669835s" totalInitContainerRuntime="28.459624ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:50.525408101 +0000 UTC m=+93.366410590" watchObservedRunningTime="2026-07-06 19:52:50.52599955 +0000 UTC m=+93.367002024" Jul 06 19:52:51.475151 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:51.475122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} Jul 06 19:52:52.482317 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:52.482278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} Jul 06 19:52:53.286266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:53.286230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:53.286438 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:53.286309 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:52:54.495780 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:54.495698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} Jul 06 19:52:54.495780 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:54.495762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} Jul 06 19:52:54.496254 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:54.495795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} Jul 06 19:52:54.496254 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:54.495824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerStarted","Data":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} Jul 06 19:52:54.528392 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:54.528347 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.490176273 podStartE2EDuration="20.528332716s" totalImagesPullingTime="12.00124157s" totalInitContainerRuntime="36.914873ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:54.5274212 +0000 UTC m=+97.368423677" watchObservedRunningTime="2026-07-06 19:52:54.528332716 +0000 UTC m=+97.369335221" Jul 06 19:52:55.202287 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:55.202255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:52:57.706510 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:57.706480 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:52:59.403755 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.403679 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-868dcf54cc-7mscx" podUID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" containerName="console" containerID="cri-o://42bdffc93f94ea9b1f502427aea6864efdf29d4fd888125f16a3eefb0502895d" gracePeriod=15 Jul 06 19:52:59.520330 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.520305 2564 generic.go:356] "Generic (PLEG): container finished" podID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" containerID="42bdffc93f94ea9b1f502427aea6864efdf29d4fd888125f16a3eefb0502895d" exitCode=0 Jul 06 19:52:59.520458 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.520381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868dcf54cc-7mscx" event={"ID":"97f90fd5-2b7b-4781-9e5b-1955fa9c8826","Type":"ContainerDied","Data":"42bdffc93f94ea9b1f502427aea6864efdf29d4fd888125f16a3eefb0502895d"} Jul 06 19:52:59.687580 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.687516 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:52:59.696976 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.696948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697027 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697090 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wcd9\" (UniqueName: \"kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697270 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697330 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697301 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config\") pod \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\" (UID: \"97f90fd5-2b7b-4781-9e5b-1955fa9c8826\") " Jul 06 19:52:59.697381 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697331 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config" (OuterVolumeSpecName: "console-config") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:52:59.697488 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697453 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca" (OuterVolumeSpecName: "service-ca") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:52:59.697616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697496 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:52:59.697616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697572 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:52:59.697616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697591 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-oauth-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:52:59.697616 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.697604 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:52:59.699458 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.699436 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:52:59.699550 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.699490 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9" (OuterVolumeSpecName: "kube-api-access-2wcd9") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "kube-api-access-2wcd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:52:59.699630 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.699614 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97f90fd5-2b7b-4781-9e5b-1955fa9c8826" (UID: "97f90fd5-2b7b-4781-9e5b-1955fa9c8826"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:52:59.798100 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.798066 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:52:59.798100 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.798101 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wcd9\" (UniqueName: \"kubernetes.io/projected/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-kube-api-access-2wcd9\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:52:59.798316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:52:59.798116 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97f90fd5-2b7b-4781-9e5b-1955fa9c8826-console-oauth-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:00.526349 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:00.526316 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868dcf54cc-7mscx" Jul 06 19:53:00.526349 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:00.526322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868dcf54cc-7mscx" event={"ID":"97f90fd5-2b7b-4781-9e5b-1955fa9c8826","Type":"ContainerDied","Data":"392f024e0cabfad569e3fc5381abc21d168e7a503b84aa4a1a797eb6b1de3df9"} Jul 06 19:53:00.533665 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:00.533642 2564 scope.go:117] "RemoveContainer" containerID="42bdffc93f94ea9b1f502427aea6864efdf29d4fd888125f16a3eefb0502895d" Jul 06 19:53:00.541568 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:00.541549 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:53:00.547221 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:00.547204 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-868dcf54cc-7mscx"] Jul 06 19:53:01.712259 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:01.712229 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" path="/var/lib/kubelet/pods/97f90fd5-2b7b-4781-9e5b-1955fa9c8826/volumes" Jul 06 19:53:12.390544 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:12.390514 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zlngt" Jul 06 19:53:13.292108 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:13.292080 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:53:13.295922 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:13.295896 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85f5fbfd6d-48sr6" Jul 06 19:53:22.736451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:22.736250 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f55f66696-4qb57" podUID="3bc68077-549f-45c4-ab99-9b47aa300e05" containerName="console" containerID="cri-o://e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605" gracePeriod=15 Jul 06 19:53:23.036007 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.035980 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:53:23.082286 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082332 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082509 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082564 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082521 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwmn4\" (UniqueName: \"kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082653 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082559 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082794 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082794 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082712 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert\") pod \"3bc68077-549f-45c4-ab99-9b47aa300e05\" (UID: \"3bc68077-549f-45c4-ab99-9b47aa300e05\") " Jul 06 19:53:23.082794 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082733 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:23.082794 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082761 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca" (OuterVolumeSpecName: "service-ca") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:23.083044 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.082990 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config" (OuterVolumeSpecName: "console-config") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:23.083106 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.083049 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.083106 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.083062 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.083106 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.083071 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-console-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.083309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.083242 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:23.084735 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.084710 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4" (OuterVolumeSpecName: "kube-api-access-mwmn4") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "kube-api-access-mwmn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:23.085235 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.085213 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:23.085337 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.085314 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3bc68077-549f-45c4-ab99-9b47aa300e05" (UID: "3bc68077-549f-45c4-ab99-9b47aa300e05"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:23.184097 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.184071 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.184097 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.184094 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmn4\" (UniqueName: \"kubernetes.io/projected/3bc68077-549f-45c4-ab99-9b47aa300e05-kube-api-access-mwmn4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.184281 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.184110 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3bc68077-549f-45c4-ab99-9b47aa300e05-console-oauth-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.184281 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.184120 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3bc68077-549f-45c4-ab99-9b47aa300e05-oauth-serving-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:23.627216 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.627178 2564 generic.go:356] "Generic (PLEG): container finished" podID="3bc68077-549f-45c4-ab99-9b47aa300e05" containerID="e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605" exitCode=0 Jul 06 19:53:23.627395 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.627226 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f55f66696-4qb57" Jul 06 19:53:23.627395 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.627260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f55f66696-4qb57" event={"ID":"3bc68077-549f-45c4-ab99-9b47aa300e05","Type":"ContainerDied","Data":"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605"} Jul 06 19:53:23.627395 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.627358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f55f66696-4qb57" event={"ID":"3bc68077-549f-45c4-ab99-9b47aa300e05","Type":"ContainerDied","Data":"aa7caa25f2735c2726d5593fbae1cd6c7bcd20aee1573a6196f94bdb9d4bb550"} Jul 06 19:53:23.627395 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.627386 2564 scope.go:117] "RemoveContainer" containerID="e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605" Jul 06 19:53:23.637195 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.637171 2564 scope.go:117] "RemoveContainer" containerID="e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605" Jul 06 19:53:23.637515 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:23.637490 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605\": container with ID starting with e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605 not found: ID does not exist" containerID="e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605" Jul 06 19:53:23.637596 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.637521 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605"} err="failed to get container status \"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605\": rpc error: code = NotFound desc = could not find container \"e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605\": container with ID starting with e63207f916c6e93ee6503468183c3bf85b7f1ebffb68bf9f1752ac7b893ae605 not found: ID does not exist" Jul 06 19:53:23.649165 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.649138 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:53:23.652401 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.652381 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f55f66696-4qb57"] Jul 06 19:53:23.713704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:23.713668 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bc68077-549f-45c4-ab99-9b47aa300e05" path="/var/lib/kubelet/pods/3bc68077-549f-45c4-ab99-9b47aa300e05/volumes" Jul 06 19:53:35.203335 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:35.203292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:35.221414 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:35.221394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:35.690573 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:35.690549 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:48.943870 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.943820 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944166 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="alertmanager" containerID="cri-o://d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44" gracePeriod=120 Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944224 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-web" containerID="cri-o://32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c" gracePeriod=120 Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944225 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-metric" containerID="cri-o://d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0" gracePeriod=120 Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944250 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="config-reloader" containerID="cri-o://28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc" gracePeriod=120 Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944297 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="prom-label-proxy" containerID="cri-o://2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7" gracePeriod=120 Jul 06 19:53:48.944411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:48.944257 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy" containerID="cri-o://2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80" gracePeriod=120 Jul 06 19:53:49.729075 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729044 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7" exitCode=0 Jul 06 19:53:49.729075 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729068 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0" exitCode=0 Jul 06 19:53:49.729075 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729074 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80" exitCode=0 Jul 06 19:53:49.729075 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729081 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc" exitCode=0 Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729086 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44" exitCode=0 Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7"} Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0"} Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80"} Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc"} Jul 06 19:53:49.729328 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:49.729244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44"} Jul 06 19:53:50.186593 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.186568 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:50.297590 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297524 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.297590 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.297766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297617 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.297766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297678 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hsd4\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.297766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297726 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.297766 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297757 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297783 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.297970 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:50.298034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.298022 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.298475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.298454 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:50.299497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.298674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle\") pod \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\" (UID: \"4e7c64ef-65d9-4119-b0c5-4f5d097a299c\") " Jul 06 19:53:50.299497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.298992 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-main-db\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.299497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.299013 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-metrics-client-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.299497 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.299456 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:50.300794 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.300760 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.301246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.301055 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume" (OuterVolumeSpecName: "config-volume") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.301246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.301205 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.301420 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.301391 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.301482 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.301471 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:50.302283 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.302258 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4" (OuterVolumeSpecName: "kube-api-access-8hsd4") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "kube-api-access-8hsd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:50.302385 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.302319 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.302385 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.302362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out" (OuterVolumeSpecName: "config-out") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:50.306625 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.306542 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.311430 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.311410 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config" (OuterVolumeSpecName: "web-config") pod "4e7c64ef-65d9-4119-b0c5-4f5d097a299c" (UID: "4e7c64ef-65d9-4119-b0c5-4f5d097a299c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:50.399800 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399777 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-web-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399800 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399799 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399810 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-cluster-tls-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399819 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399827 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399836 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-config-out\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399843 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-main-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399851 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399880 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-tls-assets\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399892 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.399966 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.399902 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsd4\" (UniqueName: \"kubernetes.io/projected/4e7c64ef-65d9-4119-b0c5-4f5d097a299c-kube-api-access-8hsd4\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:50.736107 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.736082 2564 generic.go:356] "Generic (PLEG): container finished" podID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerID="32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c" exitCode=0 Jul 06 19:53:50.736246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.736149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c"} Jul 06 19:53:50.736246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.736156 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:50.736246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.736198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e7c64ef-65d9-4119-b0c5-4f5d097a299c","Type":"ContainerDied","Data":"c9f61b4985959ac018e84c77e2e92778a66f19ad8b3f7558bd1d8ccd502bf366"} Jul 06 19:53:50.736246 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.736233 2564 scope.go:117] "RemoveContainer" containerID="2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7" Jul 06 19:53:50.743795 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.743778 2564 scope.go:117] "RemoveContainer" containerID="d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0" Jul 06 19:53:50.750586 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.750569 2564 scope.go:117] "RemoveContainer" containerID="2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80" Jul 06 19:53:50.756876 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.756847 2564 scope.go:117] "RemoveContainer" containerID="32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c" Jul 06 19:53:50.761522 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.761501 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:50.764288 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.764274 2564 scope.go:117] "RemoveContainer" containerID="28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc" Jul 06 19:53:50.767804 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.767784 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:50.771502 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.771479 2564 scope.go:117] "RemoveContainer" containerID="d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44" Jul 06 19:53:50.777983 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.777964 2564 scope.go:117] "RemoveContainer" containerID="9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa" Jul 06 19:53:50.784353 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.784329 2564 scope.go:117] "RemoveContainer" containerID="2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7" Jul 06 19:53:50.784569 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.784551 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7\": container with ID starting with 2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7 not found: ID does not exist" containerID="2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7" Jul 06 19:53:50.784615 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.784578 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7"} err="failed to get container status \"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7\": rpc error: code = NotFound desc = could not find container \"2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7\": container with ID starting with 2f71131fa36c52aecc2bb3d57bfe935b508ec1bf602acbdbe083a53a68ba10d7 not found: ID does not exist" Jul 06 19:53:50.784615 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.784596 2564 scope.go:117] "RemoveContainer" containerID="d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0" Jul 06 19:53:50.784812 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.784797 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0\": container with ID starting with d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0 not found: ID does not exist" containerID="d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0" Jul 06 19:53:50.784852 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.784817 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0"} err="failed to get container status \"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0\": rpc error: code = NotFound desc = could not find container \"d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0\": container with ID starting with d2dcad2ea8acd0ed4b43939bde3d56decf8f2b10d4249d8d13dfe8637e08fdc0 not found: ID does not exist" Jul 06 19:53:50.784852 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.784835 2564 scope.go:117] "RemoveContainer" containerID="2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80" Jul 06 19:53:50.785041 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.785024 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80\": container with ID starting with 2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80 not found: ID does not exist" containerID="2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80" Jul 06 19:53:50.785078 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785045 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80"} err="failed to get container status \"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80\": rpc error: code = NotFound desc = could not find container \"2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80\": container with ID starting with 2f2dcba053d9f6cb70ebf42ec5faa985c0a2a97d45fd54318f679ba7a7452d80 not found: ID does not exist" Jul 06 19:53:50.785078 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785057 2564 scope.go:117] "RemoveContainer" containerID="32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c" Jul 06 19:53:50.785227 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.785213 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c\": container with ID starting with 32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c not found: ID does not exist" containerID="32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c" Jul 06 19:53:50.785266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785229 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c"} err="failed to get container status \"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c\": rpc error: code = NotFound desc = could not find container \"32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c\": container with ID starting with 32a17d3dd98d70e9ce10f1a5a6dace9ed16dce8aaee2152e435109df4d70339c not found: ID does not exist" Jul 06 19:53:50.785266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785241 2564 scope.go:117] "RemoveContainer" containerID="28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc" Jul 06 19:53:50.785440 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.785427 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc\": container with ID starting with 28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc not found: ID does not exist" containerID="28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc" Jul 06 19:53:50.785477 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785442 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc"} err="failed to get container status \"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc\": rpc error: code = NotFound desc = could not find container \"28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc\": container with ID starting with 28cb8d1768f1996c2c5e4f7344b3653bc052c741d1597318a3dfb1ceec2523cc not found: ID does not exist" Jul 06 19:53:50.785477 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785453 2564 scope.go:117] "RemoveContainer" containerID="d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44" Jul 06 19:53:50.785638 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.785620 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44\": container with ID starting with d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44 not found: ID does not exist" containerID="d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44" Jul 06 19:53:50.785704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785646 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44"} err="failed to get container status \"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44\": rpc error: code = NotFound desc = could not find container \"d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44\": container with ID starting with d7ab6c0250c6a81a35169db78f1da0d25047a65754f6961d34bc204571a2bf44 not found: ID does not exist" Jul 06 19:53:50.785704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785666 2564 scope.go:117] "RemoveContainer" containerID="9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa" Jul 06 19:53:50.785916 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:50.785849 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa\": container with ID starting with 9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa not found: ID does not exist" containerID="9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa" Jul 06 19:53:50.785916 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.785891 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa"} err="failed to get container status \"9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa\": rpc error: code = NotFound desc = could not find container \"9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa\": container with ID starting with 9687950f1edb9f8ec90a2e30c742d2f6fe07bcfa1bd1b52030cc9e5edffc1baa not found: ID does not exist" Jul 06 19:53:50.810544 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810521 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:50.810806 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810792 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="config-reloader" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810811 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="config-reloader" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810823 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810829 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810837 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="prom-label-proxy" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810842 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="prom-label-proxy" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810849 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="alertmanager" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810867 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="alertmanager" Jul 06 19:53:50.810872 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810874 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc68077-549f-45c4-ab99-9b47aa300e05" containerName="console" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810880 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc68077-549f-45c4-ab99-9b47aa300e05" containerName="console" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810890 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" containerName="console" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810898 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" containerName="console" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810909 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-web" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810914 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-web" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810923 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="init-config-reloader" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810928 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="init-config-reloader" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810935 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-metric" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810940 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-metric" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810987 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-metric" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.810993 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="prom-label-proxy" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811000 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="config-reloader" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811005 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811012 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f90fd5-2b7b-4781-9e5b-1955fa9c8826" containerName="console" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811017 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="alertmanager" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811022 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" containerName="kube-rbac-proxy-web" Jul 06 19:53:50.811114 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.811028 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc68077-549f-45c4-ab99-9b47aa300e05" containerName="console" Jul 06 19:53:50.814669 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.814655 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:50.822161 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.822144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 19:53:50.823950 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.823934 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 19:53:50.825844 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.825831 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 19:53:50.825978 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.825963 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 19:53:50.826024 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.826000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 19:53:50.826180 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.826169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l2chf\"" Jul 06 19:53:50.826640 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.826629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 19:53:50.835924 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.835910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 19:53:50.839786 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.839771 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 19:53:50.845823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.845798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:50.860193 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:50.860175 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 19:53:51.005085 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-config-volume\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005085 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005085 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-config-out\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005237 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005237 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005237 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-web-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005237 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgn6t\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-kube-api-access-dgn6t\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005356 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.005475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.005388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106528 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-config-volume\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-config-out\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106621 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-web-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgn6t\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-kube-api-access-dgn6t\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.106814 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.107096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.107096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.106942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.107096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.107024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.107379 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.107357 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.108726 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.108696 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.109693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.109667 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22533833-fd51-498d-8493-589796869376-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.109820 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.109727 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110228 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.109848 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22533833-fd51-498d-8493-589796869376-config-out\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110228 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110003 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110347 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110250 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110677 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-config-volume\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.110928 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.110911 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-web-config\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.111128 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.111112 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22533833-fd51-498d-8493-589796869376-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.115729 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.115709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgn6t\" (UniqueName: \"kubernetes.io/projected/22533833-fd51-498d-8493-589796869376-kube-api-access-dgn6t\") pod \"alertmanager-main-0\" (UID: \"22533833-fd51-498d-8493-589796869376\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.122465 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.122446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 19:53:51.258563 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.258478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 19:53:51.261530 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:53:51.261503 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22533833_fd51_498d_8493_589796869376.slice/crio-7d506a9d6391885632695f651c6e37b0e461109f02d54993bd08ee73836fe6b5 WatchSource:0}: Error finding container 7d506a9d6391885632695f651c6e37b0e461109f02d54993bd08ee73836fe6b5: Status 404 returned error can't find the container with id 7d506a9d6391885632695f651c6e37b0e461109f02d54993bd08ee73836fe6b5 Jul 06 19:53:51.715885 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.713241 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7c64ef-65d9-4119-b0c5-4f5d097a299c" path="/var/lib/kubelet/pods/4e7c64ef-65d9-4119-b0c5-4f5d097a299c/volumes" Jul 06 19:53:51.741347 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.741311 2564 generic.go:356] "Generic (PLEG): container finished" podID="22533833-fd51-498d-8493-589796869376" containerID="a0e493eb2d035a9374cedcba4b1ab022cec30f52cd9521de66b110e89586482a" exitCode=0 Jul 06 19:53:51.741476 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.741393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerDied","Data":"a0e493eb2d035a9374cedcba4b1ab022cec30f52cd9521de66b110e89586482a"} Jul 06 19:53:51.741476 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:51.741460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"7d506a9d6391885632695f651c6e37b0e461109f02d54993bd08ee73836fe6b5"} Jul 06 19:53:52.748104 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"3c4f0adbe88efa8a95a7e1390566bd289a464f1cd14fdc803732ed89686ae119"} Jul 06 19:53:52.748512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"5d2522ecd8085688600f2482b0e4e258742667604e7d20a94b4867a78279f6cc"} Jul 06 19:53:52.748512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"42df154d7dcfab49c342723adf842d4fbd50e77052fa9e216ba960500c1a9e4d"} Jul 06 19:53:52.748512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"b423f4c7a99eaa465be761dd4dda075b4e807a3a4a11adaa388bca4ab4a87a64"} Jul 06 19:53:52.748512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"94b2b27f4e8ecedb36fa09b4a3feb5c2d313eaba8d474888fb4c2a8b9532c336"} Jul 06 19:53:52.748512 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.748310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22533833-fd51-498d-8493-589796869376","Type":"ContainerStarted","Data":"9321c45e848b80e0bc77648108c4f278e0b89f942849b7b54c86b85d4ce2f855"} Jul 06 19:53:52.780435 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:52.780381 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.764410991 podStartE2EDuration="2.780362649s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.951658ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:52.778343859 +0000 UTC m=+155.619346357" watchObservedRunningTime="2026-07-06 19:53:52.780362649 +0000 UTC m=+155.621365121" Jul 06 19:53:53.215704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.215671 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216055 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="prometheus" containerID="cri-o://f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" gracePeriod=600 Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216074 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy" containerID="cri-o://93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" gracePeriod=600 Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216105 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" gracePeriod=600 Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216083 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="thanos-sidecar" containerID="cri-o://51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" gracePeriod=600 Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216111 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-web" containerID="cri-o://7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" gracePeriod=600 Jul 06 19:53:53.216309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.216104 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="config-reloader" containerID="cri-o://2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" gracePeriod=600 Jul 06 19:53:53.463395 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.463372 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:53.627133 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627287 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627287 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627287 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627383 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627323 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627383 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmjbm\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627473 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627489 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627499 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627527 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627629 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627724 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627748 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627777 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627814 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.627958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.627881 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle\") pod \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\" (UID: \"e0d4232e-ab3d-4f03-ac95-8186252efcf0\") " Jul 06 19:53:53.628309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.628168 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.628309 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.628186 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.628737 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.628486 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.628737 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.628594 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.629130 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.628954 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.629237 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.629211 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.629825 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.629798 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.630131 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.630089 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm" (OuterVolumeSpecName: "kube-api-access-tmjbm") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "kube-api-access-tmjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:53.630222 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.630161 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:53:53.630528 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.630498 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.630600 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.630581 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config" (OuterVolumeSpecName: "config") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.631000 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.630976 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:53:53.631334 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.631310 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:53.632194 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.632139 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.632420 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.632392 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.633767 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.633738 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out" (OuterVolumeSpecName: "config-out") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:53:53.634092 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.634068 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.634182 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.634163 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.634335 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.634311 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.642749 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.642726 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config" (OuterVolumeSpecName: "web-config") pod "e0d4232e-ab3d-4f03-ac95-8186252efcf0" (UID: "e0d4232e-ab3d-4f03-ac95-8186252efcf0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:53:53.729257 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729231 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-web-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729257 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729254 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729263 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729274 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-metrics-client-certs\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729283 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729292 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729303 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729312 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-kube-rbac-proxy\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729320 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729330 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-db\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729337 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjbm\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-kube-api-access-tmjbm\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729345 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729354 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729363 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4232e-ab3d-4f03-ac95-8186252efcf0-configmap-metrics-client-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729372 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0d4232e-ab3d-4f03-ac95-8186252efcf0-tls-assets\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729380 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-secret-grpc-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729387 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729387 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0d4232e-ab3d-4f03-ac95-8186252efcf0-config-out\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.729791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.729395 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0d4232e-ab3d-4f03-ac95-8186252efcf0-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:53:53.755556 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755525 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" exitCode=0 Jul 06 19:53:53.755556 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755549 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" exitCode=0 Jul 06 19:53:53.755556 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755556 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" exitCode=0 Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755562 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" exitCode=0 Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755567 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" exitCode=0 Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755572 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" exitCode=0 Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755612 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755879 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} Jul 06 19:53:53.756025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.755943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0d4232e-ab3d-4f03-ac95-8186252efcf0","Type":"ContainerDied","Data":"99c732dcb79fbd5b93d8965cf7a098bfaa0bc03232a49054463126d3a4409d1e"} Jul 06 19:53:53.763958 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.763941 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.773809 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.771452 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.779141 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.779124 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.785445 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.785430 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.792007 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.791993 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.794004 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.793984 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:53.796842 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.796820 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:53.799286 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.799271 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.805426 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.805411 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.805689 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.805673 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.805742 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.805694 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.805742 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.805709 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.806013 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.805996 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.806071 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806016 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.806071 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806028 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.806260 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.806243 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.806300 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806264 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.806300 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806276 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.806475 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.806460 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.806513 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806479 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.806513 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806491 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.806701 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.806686 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.806738 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806704 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.806738 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806715 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.806910 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.806894 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.806976 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806913 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.806976 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.806923 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.807145 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:53:53.807127 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.807198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807148 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.807198 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807164 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.807379 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807364 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.807422 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807379 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.807576 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807562 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.807624 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807576 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.807748 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807731 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.807782 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807752 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.807954 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807938 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.808005 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.807956 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.808124 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808108 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.808164 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808124 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.808294 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808274 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.808330 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808295 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.808478 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808459 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.808513 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808481 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.808669 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808655 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.808709 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808669 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.808901 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808885 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.808901 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.808900 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.809091 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809068 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.809175 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809088 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.809329 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809313 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.809375 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809330 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.809548 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809532 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.809594 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809548 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.809733 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809718 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.809774 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809733 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.809922 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809908 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.809963 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.809922 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.810106 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810088 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.810189 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810108 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.810346 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810328 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.810389 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810348 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.810558 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810542 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.810599 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810559 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.810763 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810746 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.810842 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.810765 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.811019 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811002 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.811076 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811020 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.811201 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811183 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.811249 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811203 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.811425 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811408 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.811483 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811424 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.811632 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811616 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.811681 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811631 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.811826 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811811 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.811890 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.811826 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.812059 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812040 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.812096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812059 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.812266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812250 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.812305 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812265 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.812451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812435 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.812451 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812450 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.812617 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812601 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.812658 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812617 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.812826 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812808 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.812893 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.812826 2564 scope.go:117] "RemoveContainer" containerID="b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5" Jul 06 19:53:53.813019 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5"} err="failed to get container status \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": rpc error: code = NotFound desc = could not find container \"b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5\": container with ID starting with b188fd7668030118c926d4e1d47547630d84228c2d2baac8d08a780129d2dbb5 not found: ID does not exist" Jul 06 19:53:53.813062 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813020 2564 scope.go:117] "RemoveContainer" containerID="93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891" Jul 06 19:53:53.813213 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813198 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891"} err="failed to get container status \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": rpc error: code = NotFound desc = could not find container \"93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891\": container with ID starting with 93119c6b2b9a9370faa0207c9bb73a678ecfa7227f7d0474ef6eb96f71af2891 not found: ID does not exist" Jul 06 19:53:53.813213 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813212 2564 scope.go:117] "RemoveContainer" containerID="7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76" Jul 06 19:53:53.813410 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813396 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76"} err="failed to get container status \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": rpc error: code = NotFound desc = could not find container \"7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76\": container with ID starting with 7aa1a3eabd01c66a597c98e5aa4d6ba4dca2f62bfcf1b3a767e209e8b0f56e76 not found: ID does not exist" Jul 06 19:53:53.813410 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813409 2564 scope.go:117] "RemoveContainer" containerID="51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37" Jul 06 19:53:53.813579 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813564 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37"} err="failed to get container status \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": rpc error: code = NotFound desc = could not find container \"51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37\": container with ID starting with 51bb6681707de2db523b2de2cd295bbbcfd2521e774df46c444692446728cc37 not found: ID does not exist" Jul 06 19:53:53.813623 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813579 2564 scope.go:117] "RemoveContainer" containerID="2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149" Jul 06 19:53:53.813752 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813732 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149"} err="failed to get container status \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": rpc error: code = NotFound desc = could not find container \"2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149\": container with ID starting with 2c2edc8e25c06229d2217da1d6cb6c738668e9c7cab56323b1cdcc55829a5149 not found: ID does not exist" Jul 06 19:53:53.813818 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813755 2564 scope.go:117] "RemoveContainer" containerID="f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0" Jul 06 19:53:53.814011 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.813995 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0"} err="failed to get container status \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": rpc error: code = NotFound desc = could not find container \"f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0\": container with ID starting with f98c5fd1645802e1f5ae89b2da2de29f8c0309cc74fa93e93273e26f757a43e0 not found: ID does not exist" Jul 06 19:53:53.814055 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.814012 2564 scope.go:117] "RemoveContainer" containerID="67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0" Jul 06 19:53:53.814212 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.814184 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0"} err="failed to get container status \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": rpc error: code = NotFound desc = could not find container \"67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0\": container with ID starting with 67300fab0f107e485209c95c3b492ec4f0236ce4b62dde4cf3ba6f84819398f0 not found: ID does not exist" Jul 06 19:53:53.863364 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863342 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:53.863606 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863595 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="thanos-sidecar" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863608 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="thanos-sidecar" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863618 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-thanos" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863625 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-thanos" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863631 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863637 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863647 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="init-config-reloader" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863652 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="init-config-reloader" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863659 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="prometheus" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863664 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="prometheus" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863670 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="config-reloader" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863674 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="config-reloader" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863682 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-web" Jul 06 19:53:53.863694 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863687 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-web" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863729 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-web" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863737 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863742 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="config-reloader" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863749 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="kube-rbac-proxy-thanos" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863754 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="prometheus" Jul 06 19:53:53.864096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.863760 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" containerName="thanos-sidecar" Jul 06 19:53:53.867537 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.867523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:53.898653 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:53:53.898653 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:53:53.898795 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898679 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:53:53.898848 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:53:53.898913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898882 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:53:53.898913 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898898 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:53:53.899016 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.898947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:53:53.899016 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:53:53.899184 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:53:53.899307 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:53:53.899341 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:53:53.899411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-frjqx\"" Jul 06 19:53:53.899516 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.899503 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:53:53.903283 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.903249 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:53:53.909505 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.909488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:53:53.912152 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.912137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5466k88mgb2kl\"" Jul 06 19:53:53.929696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:53.929673 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:54.032050 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032174 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032174 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032174 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032299 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032299 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032299 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032299 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032299 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bb9h\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-kube-api-access-6bb9h\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.032693 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.032677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.133797 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.133768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.133797 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.133796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134049 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.133811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bb9h\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-kube-api-access-6bb9h\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134049 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.133828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134049 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134211 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134211 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134211 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134211 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134371 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134371 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134438 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134484 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134526 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134563 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134563 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134629 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134629 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134642 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134698 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134810 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.134953 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.134935 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.135443 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.135423 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.135655 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.135559 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.136134 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.136110 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.136665 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.136637 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.136959 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.136936 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.137447 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.137422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138052 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138193 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138169 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138288 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138268 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138351 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138737 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138716 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.138970 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.138950 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.139290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.139274 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.140429 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.140409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8740e8d-7981-47fc-bcc9-6745f3e84fca-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.140504 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.140403 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.140504 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.140453 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8740e8d-7981-47fc-bcc9-6745f3e84fca-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.144475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.144456 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bb9h\" (UniqueName: \"kubernetes.io/projected/d8740e8d-7981-47fc-bcc9-6745f3e84fca-kube-api-access-6bb9h\") pod \"prometheus-k8s-0\" (UID: \"d8740e8d-7981-47fc-bcc9-6745f3e84fca\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.175626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.175581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:53:54.321067 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.321039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:53:54.324893 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:53:54.324841 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8740e8d_7981_47fc_bcc9_6745f3e84fca.slice/crio-8338955a687e1fe26934e81576366654b591e1f3b3c036f3b698f876479ba176 WatchSource:0}: Error finding container 8338955a687e1fe26934e81576366654b591e1f3b3c036f3b698f876479ba176: Status 404 returned error can't find the container with id 8338955a687e1fe26934e81576366654b591e1f3b3c036f3b698f876479ba176 Jul 06 19:53:54.760384 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.760354 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8740e8d-7981-47fc-bcc9-6745f3e84fca" containerID="7af63e3d0f587a050c2ce13c07c584e0ead81d78f22b8fa3fefdfa6866e7d62d" exitCode=0 Jul 06 19:53:54.760760 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.760435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerDied","Data":"7af63e3d0f587a050c2ce13c07c584e0ead81d78f22b8fa3fefdfa6866e7d62d"} Jul 06 19:53:54.760760 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:54.760503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"8338955a687e1fe26934e81576366654b591e1f3b3c036f3b698f876479ba176"} Jul 06 19:53:55.716493 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.716462 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d4232e-ab3d-4f03-ac95-8186252efcf0" path="/var/lib/kubelet/pods/e0d4232e-ab3d-4f03-ac95-8186252efcf0/volumes" Jul 06 19:53:55.768179 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768153 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"4be230da59422fb90ec40bd0c8893cded0cf2128e470803550c173db9d3409ea"} Jul 06 19:53:55.768496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"c30b8d2dd273669a49c3e1b303b31592a06f5ed4cecc3e4e0d046bad025970a8"} Jul 06 19:53:55.768496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"eda7ffa37d26b4a62ef4ca4807489aaa893c58101942c639efe891977fe9581c"} Jul 06 19:53:55.768496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"ccd6d08f4732623cac60f292d23b90224998eb2031108269e440c2ea05adf7c9"} Jul 06 19:53:55.768496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"e8a9a622a5dc4d668d70d6645fe909e34d3b53df316e3af98c82f245a0cdb1bf"} Jul 06 19:53:55.768496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.768309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8740e8d-7981-47fc-bcc9-6745f3e84fca","Type":"ContainerStarted","Data":"c53aca51cde5df25e49b34fcb5fbaeb7e30e622718eddadd08b579e878d8f35e"} Jul 06 19:53:55.800502 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:55.800394 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7832951870000002 podStartE2EDuration="2.800378303s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.083116ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:53:55.79952509 +0000 UTC m=+158.640527564" watchObservedRunningTime="2026-07-06 19:53:55.800378303 +0000 UTC m=+158.641380777" Jul 06 19:53:59.176193 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:53:59.176156 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:54:54.176495 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:54:54.176459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:54:54.191929 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:54:54.191906 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:54:55.011093 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:54:55.011068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:55:15.020375 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.020347 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-52tj5"] Jul 06 19:55:15.022936 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.022914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.025254 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.025237 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:55:15.025787 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.025771 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:55:15.026098 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.026084 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m4tqt\"" Jul 06 19:55:15.033274 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.033250 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-52tj5"] Jul 06 19:55:15.073267 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.073229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjsrn\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-kube-api-access-kjsrn\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.073475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.073301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-bound-sa-token\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.173761 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.173723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-bound-sa-token\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.173993 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.173791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjsrn\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-kube-api-access-kjsrn\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.185760 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.185730 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-bound-sa-token\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.185973 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.185953 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjsrn\" (UniqueName: \"kubernetes.io/projected/bd3b37e0-5904-4c73-9a3e-8366b6d0dfec-kube-api-access-kjsrn\") pod \"cert-manager-55b4d954d5-52tj5\" (UID: \"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec\") " pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.332258 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.332164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-52tj5" Jul 06 19:55:15.473187 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:15.473160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-52tj5"] Jul 06 19:55:15.476011 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:55:15.475979 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd3b37e0_5904_4c73_9a3e_8366b6d0dfec.slice/crio-835c79030f01af20aca71ff105d27ca8cf3d73fc170db3c6a8c163c2ad48a788 WatchSource:0}: Error finding container 835c79030f01af20aca71ff105d27ca8cf3d73fc170db3c6a8c163c2ad48a788: Status 404 returned error can't find the container with id 835c79030f01af20aca71ff105d27ca8cf3d73fc170db3c6a8c163c2ad48a788 Jul 06 19:55:16.073637 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:16.073597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-52tj5" event={"ID":"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec","Type":"ContainerStarted","Data":"835c79030f01af20aca71ff105d27ca8cf3d73fc170db3c6a8c163c2ad48a788"} Jul 06 19:55:50.465759 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.465676 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg"] Jul 06 19:55:50.469316 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.469296 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.471724 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.471699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:55:50.472626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.472598 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:55:50.472626 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.472623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:55:50.472823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.472810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t6rmq\"" Jul 06 19:55:50.474638 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.474621 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:55:50.511120 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.511090 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg"] Jul 06 19:55:50.590952 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.590911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvnd\" (UniqueName: \"kubernetes.io/projected/a3ab030c-f8ed-4d14-9ef4-c2189338494f-kube-api-access-rrvnd\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.590952 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.590953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-webhook-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.591179 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.590988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.591179 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.591081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-manifests\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.591179 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.591112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-tmp\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.691691 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.691647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-manifests\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.691691 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.691691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-tmp\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.691967 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.691761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvnd\" (UniqueName: \"kubernetes.io/projected/a3ab030c-f8ed-4d14-9ef4-c2189338494f-kube-api-access-rrvnd\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.691967 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.691787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-webhook-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.691967 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.691818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.692122 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.692094 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-manifests\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.694222 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.694197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3ab030c-f8ed-4d14-9ef4-c2189338494f-tmp\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.694338 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.694279 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-webhook-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.694411 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.694394 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3ab030c-f8ed-4d14-9ef4-c2189338494f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.702453 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.702422 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvnd\" (UniqueName: \"kubernetes.io/projected/a3ab030c-f8ed-4d14-9ef4-c2189338494f-kube-api-access-rrvnd\") pod \"opendatahub-operator-controller-manager-549b57869c-bw2qg\" (UID: \"a3ab030c-f8ed-4d14-9ef4-c2189338494f\") " pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.780531 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.780432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:50.926611 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:50.926400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg"] Jul 06 19:55:50.929684 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:55:50.929657 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ab030c_f8ed_4d14_9ef4_c2189338494f.slice/crio-5be73f6c34f234d4900433b5be58bf53f793395f8e8ab55bb1d3909108131bf7 WatchSource:0}: Error finding container 5be73f6c34f234d4900433b5be58bf53f793395f8e8ab55bb1d3909108131bf7: Status 404 returned error can't find the container with id 5be73f6c34f234d4900433b5be58bf53f793395f8e8ab55bb1d3909108131bf7 Jul 06 19:55:51.216838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:51.216803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" event={"ID":"a3ab030c-f8ed-4d14-9ef4-c2189338494f","Type":"ContainerStarted","Data":"5be73f6c34f234d4900433b5be58bf53f793395f8e8ab55bb1d3909108131bf7"} Jul 06 19:55:54.232707 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:54.232667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" event={"ID":"a3ab030c-f8ed-4d14-9ef4-c2189338494f","Type":"ContainerStarted","Data":"de3ee07198d83b0293e0f4267c8183003b4f5a9674d4ad0e94ddc398bc0e7c44"} Jul 06 19:55:55.238190 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:55.238150 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3ab030c-f8ed-4d14-9ef4-c2189338494f" containerID="de3ee07198d83b0293e0f4267c8183003b4f5a9674d4ad0e94ddc398bc0e7c44" exitCode=0 Jul 06 19:55:55.238588 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:55.238239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" event={"ID":"a3ab030c-f8ed-4d14-9ef4-c2189338494f","Type":"ContainerDied","Data":"de3ee07198d83b0293e0f4267c8183003b4f5a9674d4ad0e94ddc398bc0e7c44"} Jul 06 19:55:56.244313 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:56.244270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" event={"ID":"a3ab030c-f8ed-4d14-9ef4-c2189338494f","Type":"ContainerStarted","Data":"5c07e87b9d4e8c7e456145b5f09863ae573f170ec87488fd6973221021f3c6de"} Jul 06 19:55:56.244827 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:56.244378 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:55:56.265838 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:56.265784 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" podStartSLOduration=1.773186648 podStartE2EDuration="6.265766369s" totalImagesPullingTime="3.397606028s" totalInitContainerRuntime="1.094973693s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:55:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:55:56.265189253 +0000 UTC m=+279.106191730" watchObservedRunningTime="2026-07-06 19:55:56.265766369 +0000 UTC m=+279.106768842" Jul 06 19:55:59.259151 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:59.259109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-52tj5" event={"ID":"bd3b37e0-5904-4c73-9a3e-8366b6d0dfec","Type":"ContainerStarted","Data":"f9cf925c70471ecc6c15f712de9ac9336be8728dcb835fafe6b3a734b355b763"} Jul 06 19:55:59.279464 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:55:59.279409 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-52tj5" podStartSLOduration=2.373043387 podStartE2EDuration="45.279392425s" totalImagesPullingTime="42.906349038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:55:59.277071784 +0000 UTC m=+282.118074258" watchObservedRunningTime="2026-07-06 19:55:59.279392425 +0000 UTC m=+282.120394897" Jul 06 19:56:07.251109 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:07.251074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-549b57869c-bw2qg" Jul 06 19:56:16.068121 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.068085 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4"] Jul 06 19:56:16.071554 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.071531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.075292 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075267 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:56:16.075436 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:56:16.075436 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:56:16.075436 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075267 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:56:16.075436 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075324 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jn46t\"" Jul 06 19:56:16.075804 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.075789 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:56:16.088053 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.088026 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4"] Jul 06 19:56:16.224000 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.223960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae323d21-5d59-4feb-b4b4-7c545d05a673-manager-config\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.224000 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.223995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.224231 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.224025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4krx\" (UniqueName: \"kubernetes.io/projected/ae323d21-5d59-4feb-b4b4-7c545d05a673-kube-api-access-f4krx\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.224231 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.224121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-metrics-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.325346 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.325246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae323d21-5d59-4feb-b4b4-7c545d05a673-manager-config\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.325346 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.325288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.325346 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.325316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4krx\" (UniqueName: \"kubernetes.io/projected/ae323d21-5d59-4feb-b4b4-7c545d05a673-kube-api-access-f4krx\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.325605 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.325441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-metrics-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.325963 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.325942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae323d21-5d59-4feb-b4b4-7c545d05a673-manager-config\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.328151 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.328129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-metrics-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.328247 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.328129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae323d21-5d59-4feb-b4b4-7c545d05a673-cert\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.339875 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.339832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4krx\" (UniqueName: \"kubernetes.io/projected/ae323d21-5d59-4feb-b4b4-7c545d05a673-kube-api-access-f4krx\") pod \"lws-controller-manager-c876dc4c4-jnmf4\" (UID: \"ae323d21-5d59-4feb-b4b4-7c545d05a673\") " pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.381557 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.381501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:16.521100 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:16.521066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4"] Jul 06 19:56:16.524305 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:56:16.524276 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae323d21_5d59_4feb_b4b4_7c545d05a673.slice/crio-ed0d66c3c1fef75e78f1dcd5c8eba3eda75f63c4a6a21d2604f245639ed3bb68 WatchSource:0}: Error finding container ed0d66c3c1fef75e78f1dcd5c8eba3eda75f63c4a6a21d2604f245639ed3bb68: Status 404 returned error can't find the container with id ed0d66c3c1fef75e78f1dcd5c8eba3eda75f63c4a6a21d2604f245639ed3bb68 Jul 06 19:56:17.342398 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:17.342359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" event={"ID":"ae323d21-5d59-4feb-b4b4-7c545d05a673","Type":"ContainerStarted","Data":"ed0d66c3c1fef75e78f1dcd5c8eba3eda75f63c4a6a21d2604f245639ed3bb68"} Jul 06 19:56:17.610303 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:17.610252 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:56:17.611095 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:17.611070 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 19:56:17.623663 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:17.623621 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:56:19.009253 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.009221 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt"] Jul 06 19:56:19.012796 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.012778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.015106 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.015086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:56:19.015735 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.015677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:56:19.016023 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.016001 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:56:19.016023 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.016008 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:56:19.016190 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.016095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:56:19.016338 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.016274 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m9cnf\"" Jul 06 19:56:19.016458 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.016415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:56:19.036484 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.036450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt"] Jul 06 19:56:19.151729 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.151684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.151946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.151751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.151946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.151794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3f3568-289c-4fa9-9247-3f5988173f42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.151946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.151915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.152090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.151972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlbb\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-kube-api-access-khlbb\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.152090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.152005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.152189 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.152133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253541 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253541 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3f3568-289c-4fa9-9247-3f5988173f42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.253823 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlbb\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-kube-api-access-khlbb\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.254055 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.253821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.254573 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.254537 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.256562 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.256516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.256687 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.256596 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.256687 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.256631 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.257060 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.257033 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd3f3568-289c-4fa9-9247-3f5988173f42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.263466 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.263398 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.264588 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.264555 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlbb\" (UniqueName: \"kubernetes.io/projected/dd3f3568-289c-4fa9-9247-3f5988173f42-kube-api-access-khlbb\") pod \"istiod-openshift-gateway-94bb8fbfd-gfptt\" (UID: \"dd3f3568-289c-4fa9-9247-3f5988173f42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.324444 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.324398 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:19.526448 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.526354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt"] Jul 06 19:56:19.530112 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:56:19.530066 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3f3568_289c_4fa9_9247_3f5988173f42.slice/crio-1af7a8885bb13e33dddbe43639f80e29d29da269c15772b290ef1f8126befb6f WatchSource:0}: Error finding container 1af7a8885bb13e33dddbe43639f80e29d29da269c15772b290ef1f8126befb6f: Status 404 returned error can't find the container with id 1af7a8885bb13e33dddbe43639f80e29d29da269c15772b290ef1f8126befb6f Jul 06 19:56:19.532438 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:19.532419 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:56:20.359040 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:20.359005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" event={"ID":"dd3f3568-289c-4fa9-9247-3f5988173f42","Type":"ContainerStarted","Data":"1af7a8885bb13e33dddbe43639f80e29d29da269c15772b290ef1f8126befb6f"} Jul 06 19:56:20.360528 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:20.360498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" event={"ID":"ae323d21-5d59-4feb-b4b4-7c545d05a673","Type":"ContainerStarted","Data":"2108db3841badb5f5cd115f7e4ffded0152902a06f1d3e147ba59aa69db8a823"} Jul 06 19:56:20.360667 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:20.360621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:56:20.381298 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:20.381231 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" podStartSLOduration=1.535225557 podStartE2EDuration="4.381208816s" totalImagesPullingTime="2.845983259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:20.377884838 +0000 UTC m=+303.218887305" watchObservedRunningTime="2026-07-06 19:56:20.381208816 +0000 UTC m=+303.222211290" Jul 06 19:56:25.270125 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:25.270072 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:25.270383 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:25.270158 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 19:56:25.386045 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:25.386003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" event={"ID":"dd3f3568-289c-4fa9-9247-3f5988173f42","Type":"ContainerStarted","Data":"c77335aa0be66afae8097e821ffec9b1366fd75c507ac5981d61c09a2832e02a"} Jul 06 19:56:25.386243 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:25.386144 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:25.411098 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:25.411046 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" podStartSLOduration=1.673832422 podStartE2EDuration="7.411025481s" totalImagesPullingTime="5.737193059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:25.409467946 +0000 UTC m=+308.250470446" watchObservedRunningTime="2026-07-06 19:56:25.411025481 +0000 UTC m=+308.252027955" Jul 06 19:56:26.394184 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:26.394151 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-gfptt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:56:26.394614 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:26.394211 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" podUID="dd3f3568-289c-4fa9-9247-3f5988173f42" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:56:29.392764 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:29.392730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gfptt" Jul 06 19:56:31.368417 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:56:31.368384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c876dc4c4-jnmf4" Jul 06 19:58:07.146890 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.146838 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk"] Jul 06 19:58:07.149975 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.149952 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.153374 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.153351 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:58:07.153638 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.153621 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:58:07.153745 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.153638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nsjzn\"" Jul 06 19:58:07.163355 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.163328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk"] Jul 06 19:58:07.264764 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.264723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.264984 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.264803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvq9v\" (UniqueName: \"kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.365743 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.365689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.365970 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.365816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvq9v\" (UniqueName: \"kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.366168 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.366150 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.376253 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.376228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvq9v\" (UniqueName: \"kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.460653 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.460546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:07.601028 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.600995 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk"] Jul 06 19:58:07.603958 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:58:07.603930 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f0399b_9032_43f9_81d4_d6a4f25aa6d1.slice/crio-66ce855cb98be7ebbd20a17482dae700e0f8253ee4fd8fbb36bc7f4469999c9b WatchSource:0}: Error finding container 66ce855cb98be7ebbd20a17482dae700e0f8253ee4fd8fbb36bc7f4469999c9b: Status 404 returned error can't find the container with id 66ce855cb98be7ebbd20a17482dae700e0f8253ee4fd8fbb36bc7f4469999c9b Jul 06 19:58:07.839203 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.839110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" event={"ID":"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1","Type":"ContainerStarted","Data":"66ce855cb98be7ebbd20a17482dae700e0f8253ee4fd8fbb36bc7f4469999c9b"} Jul 06 19:58:07.915016 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.914978 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk"] Jul 06 19:58:07.923893 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.923849 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk"] Jul 06 19:58:07.966479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.966448 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:07.970769 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.970750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:07.990664 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:07.990628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:08.071959 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.071906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.072127 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.071972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2kr\" (UniqueName: \"kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.172553 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.172518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.173041 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.172570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2kr\" (UniqueName: \"kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.173041 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.172889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.182831 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.182792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2kr\" (UniqueName: \"kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sbcmg\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.283527 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.283486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:08.438153 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.437992 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:08.634597 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:58:08.634562 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380f25e4_364d_47bc_be3d_9be23bae87b2.slice/crio-c5a9f3175f71963f5cf7b483dc5d536c45f33f2b1448b23ac7f5b13620a99598 WatchSource:0}: Error finding container c5a9f3175f71963f5cf7b483dc5d536c45f33f2b1448b23ac7f5b13620a99598: Status 404 returned error can't find the container with id c5a9f3175f71963f5cf7b483dc5d536c45f33f2b1448b23ac7f5b13620a99598 Jul 06 19:58:08.846186 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:08.846104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" event={"ID":"380f25e4-364d-47bc-be3d-9be23bae87b2","Type":"ContainerStarted","Data":"c5a9f3175f71963f5cf7b483dc5d536c45f33f2b1448b23ac7f5b13620a99598"} Jul 06 19:58:12.868025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:12.867950 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" containerName="manager" containerID="cri-o://c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932" gracePeriod=2 Jul 06 19:58:12.869702 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:12.869676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" event={"ID":"380f25e4-364d-47bc-be3d-9be23bae87b2","Type":"ContainerStarted","Data":"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763"} Jul 06 19:58:12.869828 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:12.869803 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:12.899311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:12.899263 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" podStartSLOduration=2.454054674 podStartE2EDuration="5.899244668s" totalImagesPullingTime="3.445189994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:58:12.896420171 +0000 UTC m=+415.737422644" watchObservedRunningTime="2026-07-06 19:58:12.899244668 +0000 UTC m=+415.740247141" Jul 06 19:58:13.121946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.121883 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:13.124182 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:58:13.124156 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:13.227390 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.227350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume\") pod \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " Jul 06 19:58:13.227570 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.227456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvq9v\" (UniqueName: \"kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v\") pod \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\" (UID: \"c5f0399b-9032-43f9-81d4-d6a4f25aa6d1\") " Jul 06 19:58:13.227630 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.227606 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" (UID: "c5f0399b-9032-43f9-81d4-d6a4f25aa6d1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:13.229696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.229672 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v" (OuterVolumeSpecName: "kube-api-access-mvq9v") pod "c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" (UID: "c5f0399b-9032-43f9-81d4-d6a4f25aa6d1"). InnerVolumeSpecName "kube-api-access-mvq9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:13.328136 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.328097 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvq9v\" (UniqueName: \"kubernetes.io/projected/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-kube-api-access-mvq9v\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:58:13.328136 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.328129 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1-extensions-socket-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:58:13.712162 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.712128 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" path="/var/lib/kubelet/pods/c5f0399b-9032-43f9-81d4-d6a4f25aa6d1/volumes" Jul 06 19:58:13.876319 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.876278 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" containerID="c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932" exitCode=2 Jul 06 19:58:13.876817 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.876339 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:13.876817 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.876377 2564 scope.go:117] "RemoveContainer" containerID="c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932" Jul 06 19:58:13.881124 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:58:13.881100 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-btrqk\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-btrqk" Jul 06 19:58:13.885586 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.885569 2564 scope.go:117] "RemoveContainer" containerID="c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932" Jul 06 19:58:13.885890 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:58:13.885872 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932\": container with ID starting with c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932 not found: ID does not exist" containerID="c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932" Jul 06 19:58:13.885955 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:13.885896 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932"} err="failed to get container status \"c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932\": rpc error: code = NotFound desc = could not find container \"c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932\": container with ID starting with c81e79c2a216ab21453efe95952c573596cf5f177beac68b451de04d15068932 not found: ID does not exist" Jul 06 19:58:23.879006 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:23.878975 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:36.356323 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.356276 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:36.356974 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.356513 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" podUID="380f25e4-364d-47bc-be3d-9be23bae87b2" containerName="manager" containerID="cri-o://13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763" gracePeriod=10 Jul 06 19:58:36.612724 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.612659 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:36.740513 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.740472 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n2kr\" (UniqueName: \"kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr\") pod \"380f25e4-364d-47bc-be3d-9be23bae87b2\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " Jul 06 19:58:36.740723 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.740560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume\") pod \"380f25e4-364d-47bc-be3d-9be23bae87b2\" (UID: \"380f25e4-364d-47bc-be3d-9be23bae87b2\") " Jul 06 19:58:36.741008 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.740981 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "380f25e4-364d-47bc-be3d-9be23bae87b2" (UID: "380f25e4-364d-47bc-be3d-9be23bae87b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:58:36.742646 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.742622 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr" (OuterVolumeSpecName: "kube-api-access-8n2kr") pod "380f25e4-364d-47bc-be3d-9be23bae87b2" (UID: "380f25e4-364d-47bc-be3d-9be23bae87b2"). InnerVolumeSpecName "kube-api-access-8n2kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:58:36.841290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.841249 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/380f25e4-364d-47bc-be3d-9be23bae87b2-extensions-socket-volume\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:58:36.841290 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.841280 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2kr\" (UniqueName: \"kubernetes.io/projected/380f25e4-364d-47bc-be3d-9be23bae87b2-kube-api-access-8n2kr\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:58:36.993266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.993170 2564 generic.go:356] "Generic (PLEG): container finished" podID="380f25e4-364d-47bc-be3d-9be23bae87b2" containerID="13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763" exitCode=0 Jul 06 19:58:36.993266 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.993223 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" Jul 06 19:58:36.993475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.993264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" event={"ID":"380f25e4-364d-47bc-be3d-9be23bae87b2","Type":"ContainerDied","Data":"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763"} Jul 06 19:58:36.993475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.993313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg" event={"ID":"380f25e4-364d-47bc-be3d-9be23bae87b2","Type":"ContainerDied","Data":"c5a9f3175f71963f5cf7b483dc5d536c45f33f2b1448b23ac7f5b13620a99598"} Jul 06 19:58:36.993475 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:36.993336 2564 scope.go:117] "RemoveContainer" containerID="13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763" Jul 06 19:58:37.002770 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:37.002752 2564 scope.go:117] "RemoveContainer" containerID="13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763" Jul 06 19:58:37.003066 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:58:37.003048 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763\": container with ID starting with 13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763 not found: ID does not exist" containerID="13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763" Jul 06 19:58:37.003149 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:37.003077 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763"} err="failed to get container status \"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763\": rpc error: code = NotFound desc = could not find container \"13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763\": container with ID starting with 13bff90ae4cb3464cbd89711c66c16db0547a32460f1e1d37fb77eefbff78763 not found: ID does not exist" Jul 06 19:58:37.014791 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:37.014761 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:37.023081 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:37.023052 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sbcmg"] Jul 06 19:58:37.712422 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:37.712382 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380f25e4-364d-47bc-be3d-9be23bae87b2" path="/var/lib/kubelet/pods/380f25e4-364d-47bc-be3d-9be23bae87b2/volumes" Jul 06 19:58:55.345680 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.345593 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.345968 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f25e4-364d-47bc-be3d-9be23bae87b2" containerName="manager" Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.345981 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f25e4-364d-47bc-be3d-9be23bae87b2" containerName="manager" Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.345994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" containerName="manager" Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.345999 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" containerName="manager" Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.346057 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f0399b-9032-43f9-81d4-d6a4f25aa6d1" containerName="manager" Jul 06 19:58:55.346088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.346066 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="380f25e4-364d-47bc-be3d-9be23bae87b2" containerName="manager" Jul 06 19:58:55.348914 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.348895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.351024 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.350999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:58:55.351024 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.350999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:58:55.351287 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.351271 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:58:55.351364 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.351287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s7x4d\"" Jul 06 19:58:55.357949 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.357923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:58:55.437817 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.437778 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:58:55.508355 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.508275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.508355 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.508349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5726\" (UniqueName: \"kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.609787 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.609753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.609980 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.609809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5726\" (UniqueName: \"kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.610446 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.610427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.618496 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.618468 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5726\" (UniqueName: \"kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726\") pod \"limitador-limitador-64c94fd64f-wtlgj\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.659459 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.659412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:55.796576 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:55.796541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:58:55.799928 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:58:55.799896 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28edbfcb_5eca_4093_979d_cb4b15122078.slice/crio-14150b54e9ddbfbf82c1bfda5867a7a6c91c15724e173dd54eecb246d8e8f531 WatchSource:0}: Error finding container 14150b54e9ddbfbf82c1bfda5867a7a6c91c15724e173dd54eecb246d8e8f531: Status 404 returned error can't find the container with id 14150b54e9ddbfbf82c1bfda5867a7a6c91c15724e173dd54eecb246d8e8f531 Jul 06 19:58:56.081658 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:56.081566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" event={"ID":"28edbfcb-5eca-4093-979d-cb4b15122078","Type":"ContainerStarted","Data":"14150b54e9ddbfbf82c1bfda5867a7a6c91c15724e173dd54eecb246d8e8f531"} Jul 06 19:58:59.098119 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:59.098009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" event={"ID":"28edbfcb-5eca-4093-979d-cb4b15122078","Type":"ContainerStarted","Data":"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08"} Jul 06 19:58:59.098483 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:59.098163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:58:59.120503 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:58:59.120443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" podStartSLOduration=1.314905306 podStartE2EDuration="4.120427879s" totalImagesPullingTime="2.805522573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:58:59.118580627 +0000 UTC m=+461.959583099" watchObservedRunningTime="2026-07-06 19:58:59.120427879 +0000 UTC m=+461.961430351" Jul 06 19:59:10.104311 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:10.104266 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:59:10.599337 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:10.599301 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:59:10.599536 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:10.599507 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" podUID="28edbfcb-5eca-4093-979d-cb4b15122078" containerName="limitador" containerID="cri-o://80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08" gracePeriod=30 Jul 06 19:59:11.150185 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.150153 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:59:11.152479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.152451 2564 generic.go:356] "Generic (PLEG): container finished" podID="28edbfcb-5eca-4093-979d-cb4b15122078" containerID="80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08" exitCode=0 Jul 06 19:59:11.152643 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.152491 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" Jul 06 19:59:11.152643 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.152530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" event={"ID":"28edbfcb-5eca-4093-979d-cb4b15122078","Type":"ContainerDied","Data":"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08"} Jul 06 19:59:11.152643 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.152586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wtlgj" event={"ID":"28edbfcb-5eca-4093-979d-cb4b15122078","Type":"ContainerDied","Data":"14150b54e9ddbfbf82c1bfda5867a7a6c91c15724e173dd54eecb246d8e8f531"} Jul 06 19:59:11.152643 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.152609 2564 scope.go:117] "RemoveContainer" containerID="80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08" Jul 06 19:59:11.161134 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.161109 2564 scope.go:117] "RemoveContainer" containerID="80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08" Jul 06 19:59:11.161451 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:59:11.161433 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08\": container with ID starting with 80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08 not found: ID does not exist" containerID="80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08" Jul 06 19:59:11.161500 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.161462 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08"} err="failed to get container status \"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08\": rpc error: code = NotFound desc = could not find container \"80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08\": container with ID starting with 80e887172938e7638419926931b626f19116313eb5827781391768ada88f5b08 not found: ID does not exist" Jul 06 19:59:11.258910 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.258839 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file\") pod \"28edbfcb-5eca-4093-979d-cb4b15122078\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " Jul 06 19:59:11.259124 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.258932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5726\" (UniqueName: \"kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726\") pod \"28edbfcb-5eca-4093-979d-cb4b15122078\" (UID: \"28edbfcb-5eca-4093-979d-cb4b15122078\") " Jul 06 19:59:11.259314 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.259287 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file" (OuterVolumeSpecName: "config-file") pod "28edbfcb-5eca-4093-979d-cb4b15122078" (UID: "28edbfcb-5eca-4093-979d-cb4b15122078"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:59:11.261025 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.261002 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726" (OuterVolumeSpecName: "kube-api-access-t5726") pod "28edbfcb-5eca-4093-979d-cb4b15122078" (UID: "28edbfcb-5eca-4093-979d-cb4b15122078"). InnerVolumeSpecName "kube-api-access-t5726". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:11.360549 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.360507 2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/28edbfcb-5eca-4093-979d-cb4b15122078-config-file\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.360549 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.360541 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5726\" (UniqueName: \"kubernetes.io/projected/28edbfcb-5eca-4093-979d-cb4b15122078-kube-api-access-t5726\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:59:11.474432 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.474396 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:59:11.478682 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.478655 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wtlgj"] Jul 06 19:59:11.713148 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:11.713066 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28edbfcb-5eca-4093-979d-cb4b15122078" path="/var/lib/kubelet/pods/28edbfcb-5eca-4093-979d-cb4b15122078/volumes" Jul 06 19:59:14.201428 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.201391 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-j7km9"] Jul 06 19:59:14.201917 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.201796 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28edbfcb-5eca-4093-979d-cb4b15122078" containerName="limitador" Jul 06 19:59:14.201917 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.201812 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28edbfcb-5eca-4093-979d-cb4b15122078" containerName="limitador" Jul 06 19:59:14.201917 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.201905 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28edbfcb-5eca-4093-979d-cb4b15122078" containerName="limitador" Jul 06 19:59:14.206576 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.206548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.210088 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.210052 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:59:14.210269 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.210101 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-thzvv\"" Jul 06 19:59:14.225229 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.225195 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-j7km9"] Jul 06 19:59:14.286217 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.286175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05aa6ab6-290a-4a93-8bcf-604557f681e0-data\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.286217 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.286223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fltb\" (UniqueName: \"kubernetes.io/projected/05aa6ab6-290a-4a93-8bcf-604557f681e0-kube-api-access-9fltb\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.387715 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.387672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fltb\" (UniqueName: \"kubernetes.io/projected/05aa6ab6-290a-4a93-8bcf-604557f681e0-kube-api-access-9fltb\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.387939 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.387782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05aa6ab6-290a-4a93-8bcf-604557f681e0-data\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.388176 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.388159 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/05aa6ab6-290a-4a93-8bcf-604557f681e0-data\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.403149 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.403113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fltb\" (UniqueName: \"kubernetes.io/projected/05aa6ab6-290a-4a93-8bcf-604557f681e0-kube-api-access-9fltb\") pod \"postgres-69fb4594cf-j7km9\" (UID: \"05aa6ab6-290a-4a93-8bcf-604557f681e0\") " pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.518220 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.518124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:14.660418 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:14.660386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-j7km9"] Jul 06 19:59:14.663684 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:59:14.663655 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aa6ab6_290a_4a93_8bcf_604557f681e0.slice/crio-43e1ced9a00b1516808d6b632c3e193f3f825155d60f687e28e7cbeb75215f48 WatchSource:0}: Error finding container 43e1ced9a00b1516808d6b632c3e193f3f825155d60f687e28e7cbeb75215f48: Status 404 returned error can't find the container with id 43e1ced9a00b1516808d6b632c3e193f3f825155d60f687e28e7cbeb75215f48 Jul 06 19:59:15.172360 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:15.172324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-j7km9" event={"ID":"05aa6ab6-290a-4a93-8bcf-604557f681e0","Type":"ContainerStarted","Data":"43e1ced9a00b1516808d6b632c3e193f3f825155d60f687e28e7cbeb75215f48"} Jul 06 19:59:26.195268 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:26.195241 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 19:59:27.230378 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:27.230338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-j7km9" event={"ID":"05aa6ab6-290a-4a93-8bcf-604557f681e0","Type":"ContainerStarted","Data":"a9d84e4cb376ab764f3da15188d3fbce63f7e5b3337e92c338a9cf9b4ad8f2ac"} Jul 06 19:59:27.230900 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:27.230478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:27.249704 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:27.249636 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-j7km9" podStartSLOduration=1.721835757 podStartE2EDuration="13.249619306s" totalImagesPullingTime="11.527783549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:27.248755539 +0000 UTC m=+490.089758073" watchObservedRunningTime="2026-07-06 19:59:27.249619306 +0000 UTC m=+490.090621805" Jul 06 19:59:33.265906 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:33.265850 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-j7km9" Jul 06 19:59:40.549403 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.549324 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:40.553790 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.553767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.556124 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.556096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 19:59:40.556546 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.556524 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 19:59:40.561725 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.561706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bvsb\"" Jul 06 19:59:40.566381 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.566351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:40.639618 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.639580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkj94\" (UniqueName: \"kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.639827 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.639648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.741152 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.741108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkj94\" (UniqueName: \"kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.741389 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.741184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.741389 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:59:40.741315 2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 19:59:40.741389 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:59:40.741383 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert podName:7dc7377a-8f44-4edb-b068-8c2d1e3f6165 nodeName:}" failed. No retries permitted until 2026-07-06 19:59:41.241363735 +0000 UTC m=+504.082366186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert") pod "maas-controller-77bff4f7d5-v4v9f" (UID: "7dc7377a-8f44-4edb-b068-8c2d1e3f6165") : secret "maas-controller-webhook-cert" not found Jul 06 19:59:40.743487 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.743459 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 19:59:40.747034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.747012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.750718 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.750692 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkj94\" (UniqueName: \"kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:40.759498 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.759464 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 19:59:40.842145 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.842038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.842383 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.842153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mrw\" (UniqueName: \"kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.943096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.943056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.943329 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.943126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mrw\" (UniqueName: \"kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.945725 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.945697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:40.951773 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:40.951733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mrw\" (UniqueName: \"kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw\") pod \"maas-controller-9459f97d5-4x99h\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:41.067549 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.067507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:41.224282 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.224243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 19:59:41.227747 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:59:41.227720 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d154a7_386e_474b_bf85_7f1c692693e5.slice/crio-dc2b04594ac234e6b2a9f4f9857addf53ffd2c5d57f34fe98a221a0285a99ef1 WatchSource:0}: Error finding container dc2b04594ac234e6b2a9f4f9857addf53ffd2c5d57f34fe98a221a0285a99ef1: Status 404 returned error can't find the container with id dc2b04594ac234e6b2a9f4f9857addf53ffd2c5d57f34fe98a221a0285a99ef1 Jul 06 19:59:41.247011 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.246972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:41.249422 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.249402 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") pod \"maas-controller-77bff4f7d5-v4v9f\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:41.295392 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.295351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerStarted","Data":"dc2b04594ac234e6b2a9f4f9857addf53ffd2c5d57f34fe98a221a0285a99ef1"} Jul 06 19:59:41.465096 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.465000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:41.605566 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:41.605521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:41.609175 ip-10-0-140-110 kubenswrapper[2564]: W0706 19:59:41.609147 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc7377a_8f44_4edb_b068_8c2d1e3f6165.slice/crio-a0454ddb584ec6ed27893a2c84896bc27c6028f8ede15fd24d9feb8b5229ed91 WatchSource:0}: Error finding container a0454ddb584ec6ed27893a2c84896bc27c6028f8ede15fd24d9feb8b5229ed91: Status 404 returned error can't find the container with id a0454ddb584ec6ed27893a2c84896bc27c6028f8ede15fd24d9feb8b5229ed91 Jul 06 19:59:42.306007 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:42.305965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" event={"ID":"7dc7377a-8f44-4edb-b068-8c2d1e3f6165","Type":"ContainerStarted","Data":"a0454ddb584ec6ed27893a2c84896bc27c6028f8ede15fd24d9feb8b5229ed91"} Jul 06 19:59:44.318123 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.318078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" event={"ID":"7dc7377a-8f44-4edb-b068-8c2d1e3f6165","Type":"ContainerStarted","Data":"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2"} Jul 06 19:59:44.318612 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.318152 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:44.319610 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.319584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerStarted","Data":"f07e9194e0e9e5f16c6755aaedcdb97a159836463403a642ba3c23a3c96d5570"} Jul 06 19:59:44.319722 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.319712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:44.341034 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.340975 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" podStartSLOduration=2.5029754 podStartE2EDuration="4.340957894s" totalImagesPullingTime="1.837982494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:44.339154459 +0000 UTC m=+507.180156934" watchObservedRunningTime="2026-07-06 19:59:44.340957894 +0000 UTC m=+507.181960366" Jul 06 19:59:44.362320 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:44.362246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9459f97d5-4x99h" podStartSLOduration=2.153381297 podStartE2EDuration="4.36221946s" totalImagesPullingTime="2.208838163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:44.360735383 +0000 UTC m=+507.201737880" watchObservedRunningTime="2026-07-06 19:59:44.36221946 +0000 UTC m=+507.203221933" Jul 06 19:59:55.329972 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:55.329939 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:55.330479 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:55.330008 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 19:59:55.399032 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:55.398986 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:55.399324 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:55.399292 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" podUID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" containerName="manager" containerID="cri-o://f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2" gracePeriod=30 Jul 06 19:59:56.186090 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.186061 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:56.290554 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.290506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkj94\" (UniqueName: \"kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94\") pod \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " Jul 06 19:59:56.290554 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.290552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") pod \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\" (UID: \"7dc7377a-8f44-4edb-b068-8c2d1e3f6165\") " Jul 06 19:59:56.292930 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.292890 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7dc7377a-8f44-4edb-b068-8c2d1e3f6165" (UID: "7dc7377a-8f44-4edb-b068-8c2d1e3f6165"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:59:56.293063 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.292940 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94" (OuterVolumeSpecName: "kube-api-access-vkj94") pod "7dc7377a-8f44-4edb-b068-8c2d1e3f6165" (UID: "7dc7377a-8f44-4edb-b068-8c2d1e3f6165"). InnerVolumeSpecName "kube-api-access-vkj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:59:56.376535 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.376496 2564 generic.go:356] "Generic (PLEG): container finished" podID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" containerID="f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2" exitCode=0 Jul 06 19:59:56.376971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.376545 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" Jul 06 19:59:56.376971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.376546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" event={"ID":"7dc7377a-8f44-4edb-b068-8c2d1e3f6165","Type":"ContainerDied","Data":"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2"} Jul 06 19:59:56.376971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.376681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bff4f7d5-v4v9f" event={"ID":"7dc7377a-8f44-4edb-b068-8c2d1e3f6165","Type":"ContainerDied","Data":"a0454ddb584ec6ed27893a2c84896bc27c6028f8ede15fd24d9feb8b5229ed91"} Jul 06 19:59:56.376971 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.376712 2564 scope.go:117] "RemoveContainer" containerID="f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2" Jul 06 19:59:56.385946 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.385928 2564 scope.go:117] "RemoveContainer" containerID="f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2" Jul 06 19:59:56.386232 ip-10-0-140-110 kubenswrapper[2564]: E0706 19:59:56.386214 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2\": container with ID starting with f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2 not found: ID does not exist" containerID="f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2" Jul 06 19:59:56.386300 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.386246 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2"} err="failed to get container status \"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2\": rpc error: code = NotFound desc = could not find container \"f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2\": container with ID starting with f3b9e716328ac1c3bcd3181ba64074d95dc4b0dc9b155cec1f5802d0d685d6d2 not found: ID does not exist" Jul 06 19:59:56.391409 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.391387 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkj94\" (UniqueName: \"kubernetes.io/projected/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-kube-api-access-vkj94\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:59:56.391409 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.391410 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7dc7377a-8f44-4edb-b068-8c2d1e3f6165-webhook-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 19:59:56.397907 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.397883 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:56.402147 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:56.402124 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77bff4f7d5-v4v9f"] Jul 06 19:59:57.712696 ip-10-0-140-110 kubenswrapper[2564]: I0706 19:59:57.712661 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" path="/var/lib/kubelet/pods/7dc7377a-8f44-4edb-b068-8c2d1e3f6165/volumes" Jul 06 20:00:15.896922 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.896834 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-68cb9b94f4-8bs7f"] Jul 06 20:00:15.897438 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.897214 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" containerName="manager" Jul 06 20:00:15.897438 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.897226 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" containerName="manager" Jul 06 20:00:15.897438 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.897284 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc7377a-8f44-4edb-b068-8c2d1e3f6165" containerName="manager" Jul 06 20:00:15.900510 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.900490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:15.903615 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.903591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 20:00:15.914703 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.914676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68cb9b94f4-8bs7f"] Jul 06 20:00:15.974642 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.974604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7900e681-d48f-46a3-8e62-0725249f6e7f-openshift-service-ca\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:15.974642 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.974643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:15.974876 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:15.974685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkht\" (UniqueName: \"kubernetes.io/projected/7900e681-d48f-46a3-8e62-0725249f6e7f-kube-api-access-zhkht\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.076171 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.076130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkht\" (UniqueName: \"kubernetes.io/projected/7900e681-d48f-46a3-8e62-0725249f6e7f-kube-api-access-zhkht\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.076361 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.076222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7900e681-d48f-46a3-8e62-0725249f6e7f-openshift-service-ca\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.076361 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.076248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.076438 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:00:16.076360 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 06 20:00:16.076438 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:00:16.076417 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls podName:7900e681-d48f-46a3-8e62-0725249f6e7f nodeName:}" failed. No retries permitted until 2026-07-06 20:00:16.576397823 +0000 UTC m=+539.417400275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls") pod "maas-api-68cb9b94f4-8bs7f" (UID: "7900e681-d48f-46a3-8e62-0725249f6e7f") : secret "maas-api-serving-cert" not found Jul 06 20:00:16.076887 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.076839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7900e681-d48f-46a3-8e62-0725249f6e7f-openshift-service-ca\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.086716 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.086681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkht\" (UniqueName: \"kubernetes.io/projected/7900e681-d48f-46a3-8e62-0725249f6e7f-kube-api-access-zhkht\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.580349 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.580313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.582911 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.582885 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7900e681-d48f-46a3-8e62-0725249f6e7f-maas-api-tls\") pod \"maas-api-68cb9b94f4-8bs7f\" (UID: \"7900e681-d48f-46a3-8e62-0725249f6e7f\") " pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.810654 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.810611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:16.948666 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:16.948639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68cb9b94f4-8bs7f"] Jul 06 20:00:16.951340 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:00:16.951312 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7900e681_d48f_46a3_8e62_0725249f6e7f.slice/crio-333272a902f537fcf81b8025c25a55b9c6cc0f1c9a89fd39541f854f99809f25 WatchSource:0}: Error finding container 333272a902f537fcf81b8025c25a55b9c6cc0f1c9a89fd39541f854f99809f25: Status 404 returned error can't find the container with id 333272a902f537fcf81b8025c25a55b9c6cc0f1c9a89fd39541f854f99809f25 Jul 06 20:00:17.474969 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:17.474921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" event={"ID":"7900e681-d48f-46a3-8e62-0725249f6e7f","Type":"ContainerStarted","Data":"333272a902f537fcf81b8025c25a55b9c6cc0f1c9a89fd39541f854f99809f25"} Jul 06 20:00:19.487214 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:19.487175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" event={"ID":"7900e681-d48f-46a3-8e62-0725249f6e7f","Type":"ContainerStarted","Data":"dddde77a74ec32190e35f35bd8be09423b39ba7bf513aabf29b32353eabf25f7"} Jul 06 20:00:19.487673 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:19.487263 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:19.509244 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:19.509190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" podStartSLOduration=2.842725607 podStartE2EDuration="4.509173937s" totalImagesPullingTime="1.66644833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:19.50613564 +0000 UTC m=+542.347138113" watchObservedRunningTime="2026-07-06 20:00:19.509173937 +0000 UTC m=+542.350176410" Jul 06 20:00:25.497210 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:25.497181 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-68cb9b94f4-8bs7f" Jul 06 20:00:31.242943 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.242903 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm"] Jul 06 20:00:31.246725 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.246708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.249557 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.249530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hqwfh\"" Jul 06 20:00:31.249705 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.249531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 20:00:31.249705 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.249585 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 20:00:31.249705 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.249532 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 20:00:31.259119 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.259090 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm"] Jul 06 20:00:31.321774 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.321731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.321774 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.321769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.321993 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.321795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.321993 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.321890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac879da-4cb3-4323-9d5e-ef9b767752a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.321993 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.321959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.322092 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.322009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xkl\" (UniqueName: \"kubernetes.io/projected/eac879da-4cb3-4323-9d5e-ef9b767752a6-kube-api-access-29xkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423363 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xkl\" (UniqueName: \"kubernetes.io/projected/eac879da-4cb3-4323-9d5e-ef9b767752a6-kube-api-access-29xkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423537 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423537 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423537 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423660 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac879da-4cb3-4323-9d5e-ef9b767752a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423660 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423865 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423845 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.423978 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423949 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.424094 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.423955 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.425697 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.425675 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac879da-4cb3-4323-9d5e-ef9b767752a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.426045 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.426024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac879da-4cb3-4323-9d5e-ef9b767752a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.432533 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.432508 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xkl\" (UniqueName: \"kubernetes.io/projected/eac879da-4cb3-4323-9d5e-ef9b767752a6-kube-api-access-29xkl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm\" (UID: \"eac879da-4cb3-4323-9d5e-ef9b767752a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.556971 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.556885 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:31.693421 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:31.693387 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm"] Jul 06 20:00:31.697094 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:00:31.697054 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac879da_4cb3_4323_9d5e_ef9b767752a6.slice/crio-5b762f1826af0340b112e82aa954c4f2ef4163e78a32e19ab396a807a86f63a9 WatchSource:0}: Error finding container 5b762f1826af0340b112e82aa954c4f2ef4163e78a32e19ab396a807a86f63a9: Status 404 returned error can't find the container with id 5b762f1826af0340b112e82aa954c4f2ef4163e78a32e19ab396a807a86f63a9 Jul 06 20:00:32.548620 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:32.548583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" event={"ID":"eac879da-4cb3-4323-9d5e-ef9b767752a6","Type":"ContainerStarted","Data":"5b762f1826af0340b112e82aa954c4f2ef4163e78a32e19ab396a807a86f63a9"} Jul 06 20:00:41.592030 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:41.591988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" event={"ID":"eac879da-4cb3-4323-9d5e-ef9b767752a6","Type":"ContainerStarted","Data":"29273bf9448ebe13bfb9683189c5f72f25998b7d0f174d964ea72a67cc7432d3"} Jul 06 20:00:41.610921 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:41.610849 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" podStartSLOduration=1.6979435440000001 podStartE2EDuration="10.610832242s" totalImagesPullingTime="8.912888698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:00:41.609157776 +0000 UTC m=+564.450160250" watchObservedRunningTime="2026-07-06 20:00:41.610832242 +0000 UTC m=+564.451834716" Jul 06 20:00:51.557554 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:51.557510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:51.558030 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:51.557584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:51.569832 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:51.569807 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:00:51.649461 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:00:51.649433 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm" Jul 06 20:01:03.117341 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.117300 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8"] Jul 06 20:01:03.120853 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.120828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.123113 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.123089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 20:01:03.131780 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.131753 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8"] Jul 06 20:01:03.312893 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.312819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.313108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.312932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.313108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.312962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247r8\" (UniqueName: \"kubernetes.io/projected/fb168aef-c11f-4584-9018-b0b56835d475-kube-api-access-247r8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.313108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.312990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb168aef-c11f-4584-9018-b0b56835d475-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.313108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.313015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.313108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.313069 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247r8\" (UniqueName: \"kubernetes.io/projected/fb168aef-c11f-4584-9018-b0b56835d475-kube-api-access-247r8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414108 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb168aef-c11f-4584-9018-b0b56835d475-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414476 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414476 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414619 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414479 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414619 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414535 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.414699 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.414627 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.416380 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.416361 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb168aef-c11f-4584-9018-b0b56835d475-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.416640 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.416540 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb168aef-c11f-4584-9018-b0b56835d475-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.422129 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.422103 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247r8\" (UniqueName: \"kubernetes.io/projected/fb168aef-c11f-4584-9018-b0b56835d475-kube-api-access-247r8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fpwh8\" (UID: \"fb168aef-c11f-4584-9018-b0b56835d475\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.433080 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.433049 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:03.576182 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.576151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8"] Jul 06 20:01:03.579061 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:01:03.579025 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb168aef_c11f_4584_9018_b0b56835d475.slice/crio-224a6634745acd7a6bec74a4040dfad9c01b1c8faa34d08e36c60c54a736b374 WatchSource:0}: Error finding container 224a6634745acd7a6bec74a4040dfad9c01b1c8faa34d08e36c60c54a736b374: Status 404 returned error can't find the container with id 224a6634745acd7a6bec74a4040dfad9c01b1c8faa34d08e36c60c54a736b374 Jul 06 20:01:03.693737 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:03.693637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" event={"ID":"fb168aef-c11f-4584-9018-b0b56835d475","Type":"ContainerStarted","Data":"224a6634745acd7a6bec74a4040dfad9c01b1c8faa34d08e36c60c54a736b374"} Jul 06 20:01:04.699952 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:04.699911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" event={"ID":"fb168aef-c11f-4584-9018-b0b56835d475","Type":"ContainerStarted","Data":"5c1ae50fd1f4e880e41c9bff1f6ddc8e1d56a266d8d70b6873a32700dd262cef"} Jul 06 20:01:04.718586 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:04.718529 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" podStartSLOduration=1.199014622 podStartE2EDuration="1.718509469s" totalImagesPullingTime="519.494847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:01:04.717254763 +0000 UTC m=+587.558257237" watchObservedRunningTime="2026-07-06 20:01:04.718509469 +0000 UTC m=+587.559511943" Jul 06 20:01:13.434137 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:13.434094 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:13.434621 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:13.434195 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:13.447107 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:13.447073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:13.753488 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:13.753409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fpwh8" Jul 06 20:01:17.648900 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:17.648841 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:01:17.656433 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:01:17.656402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:02:36.326269 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:36.326216 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-9459f97d5-4x99h" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:02:43.155324 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:43.155284 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerID="f07e9194e0e9e5f16c6755aaedcdb97a159836463403a642ba3c23a3c96d5570" exitCode=1 Jul 06 20:02:43.155727 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:43.155338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerDied","Data":"f07e9194e0e9e5f16c6755aaedcdb97a159836463403a642ba3c23a3c96d5570"} Jul 06 20:02:43.155727 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:43.155636 2564 scope.go:117] "RemoveContainer" containerID="f07e9194e0e9e5f16c6755aaedcdb97a159836463403a642ba3c23a3c96d5570" Jul 06 20:02:43.155987 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:43.155974 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:02:44.161275 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:44.161241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerStarted","Data":"87cb7e19bfa2725cdb7ad2d8d94db714f1dfcac752a0b0a6919a343634344ed1"} Jul 06 20:02:44.161712 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:44.161446 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 20:02:55.170287 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:02:55.170253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 20:03:07.877389 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.877352 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd4ff54bc-rxzb6"] Jul 06 20:03:07.881013 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.880988 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.883778 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.883755 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 20:03:07.884018 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.883996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 20:03:07.884308 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.884292 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 20:03:07.884361 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.884331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 20:03:07.884417 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.884379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 20:03:07.884807 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.884791 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zs2b6\"" Jul 06 20:03:07.889799 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.889777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 20:03:07.894077 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.894053 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd4ff54bc-rxzb6"] Jul 06 20:03:07.936394 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-trusted-ca-bundle\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936394 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5h5\" (UniqueName: \"kubernetes.io/projected/45d7c16c-c22c-43b3-9daa-5327482719d8-kube-api-access-8b5h5\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-oauth-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-service-ca\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-console-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:07.936683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:07.936675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-oauth-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.037787 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.037748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-oauth-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038026 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.037803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-service-ca\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038026 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.037919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038026 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.037954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-console-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038026 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.037985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-oauth-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038268 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.038080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-trusted-ca-bundle\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038268 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.038099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5h5\" (UniqueName: \"kubernetes.io/projected/45d7c16c-c22c-43b3-9daa-5327482719d8-kube-api-access-8b5h5\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038627 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.038600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-service-ca\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038784 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.038763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-oauth-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.038784 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.038772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-console-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.039124 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.039101 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45d7c16c-c22c-43b3-9daa-5327482719d8-trusted-ca-bundle\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.040646 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.040623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-oauth-config\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.040739 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.040683 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45d7c16c-c22c-43b3-9daa-5327482719d8-console-serving-cert\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.046719 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.046699 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5h5\" (UniqueName: \"kubernetes.io/projected/45d7c16c-c22c-43b3-9daa-5327482719d8-kube-api-access-8b5h5\") pod \"console-7cd4ff54bc-rxzb6\" (UID: \"45d7c16c-c22c-43b3-9daa-5327482719d8\") " pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.191732 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.191640 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:08.348883 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:08.348835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd4ff54bc-rxzb6"] Jul 06 20:03:08.352328 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:03:08.352296 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d7c16c_c22c_43b3_9daa_5327482719d8.slice/crio-bf75bcea2c673e1cd869f5a273c58a52ac4ecb465fe15f22e5728030471ba326 WatchSource:0}: Error finding container bf75bcea2c673e1cd869f5a273c58a52ac4ecb465fe15f22e5728030471ba326: Status 404 returned error can't find the container with id bf75bcea2c673e1cd869f5a273c58a52ac4ecb465fe15f22e5728030471ba326 Jul 06 20:03:09.280526 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:09.280485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd4ff54bc-rxzb6" event={"ID":"45d7c16c-c22c-43b3-9daa-5327482719d8","Type":"ContainerStarted","Data":"9b4a766606fbbf1089ddb9c63db2f23dbbad54132aba5efc224ba8102dca77a3"} Jul 06 20:03:09.280989 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:09.280548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd4ff54bc-rxzb6" event={"ID":"45d7c16c-c22c-43b3-9daa-5327482719d8","Type":"ContainerStarted","Data":"bf75bcea2c673e1cd869f5a273c58a52ac4ecb465fe15f22e5728030471ba326"} Jul 06 20:03:09.458039 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:09.457917 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cd4ff54bc-rxzb6" podStartSLOduration=2.457898999 podStartE2EDuration="2.457898999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:03:09.456250988 +0000 UTC m=+712.297253462" watchObservedRunningTime="2026-07-06 20:03:09.457898999 +0000 UTC m=+712.298901472" Jul 06 20:03:18.192029 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:18.191991 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:18.192496 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:18.192068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:18.197046 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:18.197021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:03:18.325750 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:03:18.325722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd4ff54bc-rxzb6" Jul 06 20:04:00.349543 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.349495 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 20:04:00.350136 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.349891 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-9459f97d5-4x99h" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" containerID="cri-o://87cb7e19bfa2725cdb7ad2d8d94db714f1dfcac752a0b0a6919a343634344ed1" gracePeriod=30 Jul 06 20:04:00.523606 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.523563 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerID="87cb7e19bfa2725cdb7ad2d8d94db714f1dfcac752a0b0a6919a343634344ed1" exitCode=0 Jul 06 20:04:00.523779 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.523632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerDied","Data":"87cb7e19bfa2725cdb7ad2d8d94db714f1dfcac752a0b0a6919a343634344ed1"} Jul 06 20:04:00.523779 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.523702 2564 scope.go:117] "RemoveContainer" containerID="f07e9194e0e9e5f16c6755aaedcdb97a159836463403a642ba3c23a3c96d5570" Jul 06 20:04:00.613741 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.613712 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 20:04:00.747878 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.747836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert\") pod \"c4d154a7-386e-474b-bf85-7f1c692693e5\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " Jul 06 20:04:00.748086 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.747958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mrw\" (UniqueName: \"kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw\") pod \"c4d154a7-386e-474b-bf85-7f1c692693e5\" (UID: \"c4d154a7-386e-474b-bf85-7f1c692693e5\") " Jul 06 20:04:00.749978 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.749951 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw" (OuterVolumeSpecName: "kube-api-access-68mrw") pod "c4d154a7-386e-474b-bf85-7f1c692693e5" (UID: "c4d154a7-386e-474b-bf85-7f1c692693e5"). InnerVolumeSpecName "kube-api-access-68mrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:04:00.750100 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.750074 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c4d154a7-386e-474b-bf85-7f1c692693e5" (UID: "c4d154a7-386e-474b-bf85-7f1c692693e5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:04:00.850176 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.850081 2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4d154a7-386e-474b-bf85-7f1c692693e5-webhook-cert\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:04:00.850176 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:00.850117 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mrw\" (UniqueName: \"kubernetes.io/projected/c4d154a7-386e-474b-bf85-7f1c692693e5-kube-api-access-68mrw\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:04:01.529935 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.529893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-4x99h" event={"ID":"c4d154a7-386e-474b-bf85-7f1c692693e5","Type":"ContainerDied","Data":"dc2b04594ac234e6b2a9f4f9857addf53ffd2c5d57f34fe98a221a0285a99ef1"} Jul 06 20:04:01.530411 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.529959 2564 scope.go:117] "RemoveContainer" containerID="87cb7e19bfa2725cdb7ad2d8d94db714f1dfcac752a0b0a6919a343634344ed1" Jul 06 20:04:01.530411 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.529912 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-4x99h" Jul 06 20:04:01.551736 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.551702 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 20:04:01.568517 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.568483 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9459f97d5-4x99h"] Jul 06 20:04:01.712715 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:01.712676 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" path="/var/lib/kubelet/pods/c4d154a7-386e-474b-bf85-7f1c692693e5/volumes" Jul 06 20:04:02.361177 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.361138 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9459f97d5-xmd49"] Jul 06 20:04:02.361521 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.361509 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:04:02.361571 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.361523 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:04:02.361610 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.361590 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:04:02.361610 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.361599 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:04:02.366309 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.366287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.368269 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.368247 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 20:04:02.368715 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.368699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bvsb\"" Jul 06 20:04:02.368768 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.368720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 20:04:02.375391 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.375365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9459f97d5-xmd49"] Jul 06 20:04:02.462209 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.462173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5469d566-795f-4a6d-aea8-eb8bfe112cbb-webhook-cert\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.462431 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.462302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zg6\" (UniqueName: \"kubernetes.io/projected/5469d566-795f-4a6d-aea8-eb8bfe112cbb-kube-api-access-79zg6\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.563096 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.563056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5469d566-795f-4a6d-aea8-eb8bfe112cbb-webhook-cert\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.563509 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.563131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zg6\" (UniqueName: \"kubernetes.io/projected/5469d566-795f-4a6d-aea8-eb8bfe112cbb-kube-api-access-79zg6\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.565608 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.565579 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5469d566-795f-4a6d-aea8-eb8bfe112cbb-webhook-cert\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.571739 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.571708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zg6\" (UniqueName: \"kubernetes.io/projected/5469d566-795f-4a6d-aea8-eb8bfe112cbb-kube-api-access-79zg6\") pod \"maas-controller-9459f97d5-xmd49\" (UID: \"5469d566-795f-4a6d-aea8-eb8bfe112cbb\") " pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.678177 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.678063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:02.822359 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:02.822322 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9459f97d5-xmd49"] Jul 06 20:04:02.825641 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:04:02.825604 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5469d566_795f_4a6d_aea8_eb8bfe112cbb.slice/crio-1f8c7898609f2530aee48b0bf30398c6494c178c099e9479adac8fb184a759a6 WatchSource:0}: Error finding container 1f8c7898609f2530aee48b0bf30398c6494c178c099e9479adac8fb184a759a6: Status 404 returned error can't find the container with id 1f8c7898609f2530aee48b0bf30398c6494c178c099e9479adac8fb184a759a6 Jul 06 20:04:03.542624 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:03.542588 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-xmd49" event={"ID":"5469d566-795f-4a6d-aea8-eb8bfe112cbb","Type":"ContainerStarted","Data":"ff97445a07f63cda94e10cc8caae111a9eca295ef3f7d0b5091d9ad2746c6c1c"} Jul 06 20:04:03.542830 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:03.542645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9459f97d5-xmd49" event={"ID":"5469d566-795f-4a6d-aea8-eb8bfe112cbb","Type":"ContainerStarted","Data":"1f8c7898609f2530aee48b0bf30398c6494c178c099e9479adac8fb184a759a6"} Jul 06 20:04:03.542830 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:03.542724 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:04:03.562202 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:03.562138 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9459f97d5-xmd49" podStartSLOduration=1.023674647 podStartE2EDuration="1.562120994s" totalImagesPullingTime="538.446347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:04:03.559807815 +0000 UTC m=+766.400810288" watchObservedRunningTime="2026-07-06 20:04:03.562120994 +0000 UTC m=+766.403123466" Jul 06 20:04:14.554407 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:04:14.554372 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9459f97d5-xmd49" Jul 06 20:06:17.681806 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:06:17.681728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:06:17.696049 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:06:17.696025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:11:17.717434 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:11:17.717409 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:11:17.727234 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:11:17.727210 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:14:45.549304 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:14:45.549263 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-9459f97d5-xmd49" podUID="5469d566-795f-4a6d-aea8-eb8bfe112cbb" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:14:55.549501 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:14:55.549462 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-9459f97d5-xmd49" podUID="5469d566-795f-4a6d-aea8-eb8bfe112cbb" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 20:15:01.594095 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.594063 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-8vtvn"] Jul 06 20:15:01.594497 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.594417 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:15:01.594497 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.594429 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4d154a7-386e-474b-bf85-7f1c692693e5" containerName="manager" Jul 06 20:15:01.597450 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.597434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:01.600706 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.600688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:15:01.612978 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.612953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-8vtvn"] Jul 06 20:15:01.665146 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.665111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr92n\" (UniqueName: \"kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n\") pod \"maas-api-key-cleanup-29722815-8vtvn\" (UID: \"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6\") " pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:01.766079 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.766043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr92n\" (UniqueName: \"kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n\") pod \"maas-api-key-cleanup-29722815-8vtvn\" (UID: \"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6\") " pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:01.777827 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.777803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr92n\" (UniqueName: \"kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n\") pod \"maas-api-key-cleanup-29722815-8vtvn\" (UID: \"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6\") " pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:01.908333 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:01.908311 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:02.042144 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:02.042109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722815-8vtvn"] Jul 06 20:15:02.044267 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:15:02.044239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7d4ef_0b7c_4954_8ffc_812bf9a324d6.slice/crio-a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029 WatchSource:0}: Error finding container a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029: Status 404 returned error can't find the container with id a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029 Jul 06 20:15:02.046128 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:02.046112 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:15:02.514174 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:02.514130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" event={"ID":"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6","Type":"ContainerStarted","Data":"a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029"} Jul 06 20:15:04.525403 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:04.525361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" event={"ID":"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6","Type":"ContainerStarted","Data":"a928e9ad99a1df2c180f97354131b89321507c6f9bf846a3a2a159812f796f49"} Jul 06 20:15:04.543929 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:04.543882 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" podStartSLOduration=1.404319954 podStartE2EDuration="3.543848214s" totalImagesPullingTime="2.13952826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:15:04.542416201 +0000 UTC m=+1427.383418676" watchObservedRunningTime="2026-07-06 20:15:04.543848214 +0000 UTC m=+1427.384850686" Jul 06 20:15:05.531262 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:05.531231 2564 generic.go:356] "Generic (PLEG): container finished" podID="a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6" containerID="a928e9ad99a1df2c180f97354131b89321507c6f9bf846a3a2a159812f796f49" exitCode=0 Jul 06 20:15:05.531633 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:05.531304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" event={"ID":"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6","Type":"ContainerDied","Data":"a928e9ad99a1df2c180f97354131b89321507c6f9bf846a3a2a159812f796f49"} Jul 06 20:15:06.666951 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:06.666923 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:06.700664 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:06.700637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr92n\" (UniqueName: \"kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n\") pod \"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6\" (UID: \"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6\") " Jul 06 20:15:06.702765 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:06.702728 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n" (OuterVolumeSpecName: "kube-api-access-tr92n") pod "a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6" (UID: "a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6"). InnerVolumeSpecName "kube-api-access-tr92n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:15:06.802315 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:06.802173 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr92n\" (UniqueName: \"kubernetes.io/projected/a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6-kube-api-access-tr92n\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:15:07.543256 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:07.543223 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" Jul 06 20:15:07.543439 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:07.543259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722815-8vtvn" event={"ID":"a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6","Type":"ContainerDied","Data":"a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029"} Jul 06 20:15:07.543439 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:15:07.543307 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f919b9c54c4beecab16c97de8044376e327f694beb2813ae13771cfd846029" Jul 06 20:16:17.752997 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:17.752957 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:16:17.758821 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:17.758800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:16:57.067328 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.067293 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:16:57.067780 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.067717 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6" containerName="cleanup" Jul 06 20:16:57.067780 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.067730 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6" containerName="cleanup" Jul 06 20:16:57.067876 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.067812 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6" containerName="cleanup" Jul 06 20:16:57.070501 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.070484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.074571 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.074546 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-eda21f01-serving-cert\"" Jul 06 20:16:57.074707 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.074642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:16:57.084665 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.084633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.084839 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.084694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4c9j\" (UniqueName: \"kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.084839 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.084733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.087778 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.087750 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:16:57.186225 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.186185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.186433 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.186264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4c9j\" (UniqueName: \"kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.186433 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.186308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.186433 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:16:57.186339 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-eda21f01-serving-cert: secret "maas-api-e2e-ait-eda21f01-serving-cert" not found Jul 06 20:16:57.186433 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:16:57.186432 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls podName:81488091-e0c8-4a0b-8fc4-6bf28862e91a nodeName:}" failed. No retries permitted until 2026-07-06 20:16:57.686411053 +0000 UTC m=+1540.527413505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls") pod "maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" (UID: "81488091-e0c8-4a0b-8fc4-6bf28862e91a") : secret "maas-api-e2e-ait-eda21f01-serving-cert" not found Jul 06 20:16:57.187172 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.187145 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.195735 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.195703 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4c9j\" (UniqueName: \"kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.690524 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.690478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.693222 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.693193 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") pod \"maas-api-e2e-ait-eda21f01-6746f4f45c-55r94\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:57.981005 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:57.980905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:58.129476 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:58.129439 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:16:58.133631 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:16:58.133599 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81488091_e0c8_4a0b_8fc4_6bf28862e91a.slice/crio-61b6035f1a08836075a277e63c96d5f1df3bb47053698f56855c4bb28c9413ff WatchSource:0}: Error finding container 61b6035f1a08836075a277e63c96d5f1df3bb47053698f56855c4bb28c9413ff: Status 404 returned error can't find the container with id 61b6035f1a08836075a277e63c96d5f1df3bb47053698f56855c4bb28c9413ff Jul 06 20:16:59.071118 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:59.071080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" event={"ID":"81488091-e0c8-4a0b-8fc4-6bf28862e91a","Type":"ContainerStarted","Data":"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1"} Jul 06 20:16:59.071408 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:59.071132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" event={"ID":"81488091-e0c8-4a0b-8fc4-6bf28862e91a","Type":"ContainerStarted","Data":"61b6035f1a08836075a277e63c96d5f1df3bb47053698f56855c4bb28c9413ff"} Jul 06 20:16:59.071408 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:59.071250 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:16:59.089385 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:16:59.089330 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" podStartSLOduration=1.6333278789999999 podStartE2EDuration="2.089312749s" totalImagesPullingTime="455.98487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:16:59.087902726 +0000 UTC m=+1541.928905215" watchObservedRunningTime="2026-07-06 20:16:59.089312749 +0000 UTC m=+1541.930315223" Jul 06 20:17:02.425875 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.425821 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:17:02.426383 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.426107 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" podUID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" containerName="maas-api" containerID="cri-o://2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1" gracePeriod=30 Jul 06 20:17:02.686698 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.686627 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:17:02.745354 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.745314 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca\") pod \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " Jul 06 20:17:02.745547 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.745367 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") pod \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " Jul 06 20:17:02.745547 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.745398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4c9j\" (UniqueName: \"kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j\") pod \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\" (UID: \"81488091-e0c8-4a0b-8fc4-6bf28862e91a\") " Jul 06 20:17:02.745716 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.745689 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81488091-e0c8-4a0b-8fc4-6bf28862e91a" (UID: "81488091-e0c8-4a0b-8fc4-6bf28862e91a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:02.747512 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.747482 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81488091-e0c8-4a0b-8fc4-6bf28862e91a" (UID: "81488091-e0c8-4a0b-8fc4-6bf28862e91a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:17:02.747634 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.747525 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j" (OuterVolumeSpecName: "kube-api-access-p4c9j") pod "81488091-e0c8-4a0b-8fc4-6bf28862e91a" (UID: "81488091-e0c8-4a0b-8fc4-6bf28862e91a"). InnerVolumeSpecName "kube-api-access-p4c9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:17:02.846289 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.846246 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81488091-e0c8-4a0b-8fc4-6bf28862e91a-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:02.846289 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.846279 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81488091-e0c8-4a0b-8fc4-6bf28862e91a-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:02.846289 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:02.846289 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4c9j\" (UniqueName: \"kubernetes.io/projected/81488091-e0c8-4a0b-8fc4-6bf28862e91a-kube-api-access-p4c9j\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:03.091874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.091765 2564 generic.go:356] "Generic (PLEG): container finished" podID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" containerID="2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1" exitCode=0 Jul 06 20:17:03.091874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.091813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" event={"ID":"81488091-e0c8-4a0b-8fc4-6bf28862e91a","Type":"ContainerDied","Data":"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1"} Jul 06 20:17:03.091874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.091816 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" Jul 06 20:17:03.091874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.091870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94" event={"ID":"81488091-e0c8-4a0b-8fc4-6bf28862e91a","Type":"ContainerDied","Data":"61b6035f1a08836075a277e63c96d5f1df3bb47053698f56855c4bb28c9413ff"} Jul 06 20:17:03.092175 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.091893 2564 scope.go:117] "RemoveContainer" containerID="2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1" Jul 06 20:17:03.101251 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.101229 2564 scope.go:117] "RemoveContainer" containerID="2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1" Jul 06 20:17:03.101529 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:03.101502 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1\": container with ID starting with 2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1 not found: ID does not exist" containerID="2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1" Jul 06 20:17:03.101636 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.101533 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1"} err="failed to get container status \"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1\": rpc error: code = NotFound desc = could not find container \"2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1\": container with ID starting with 2400051a5c90031a7b297ee874961e3e41c8138e366ddd101bd50ea2edde8de1 not found: ID does not exist" Jul 06 20:17:03.111617 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.111575 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:17:03.118294 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.118262 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-eda21f01-6746f4f45c-55r94"] Jul 06 20:17:03.713792 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:03.713747 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" path="/var/lib/kubelet/pods/81488091-e0c8-4a0b-8fc4-6bf28862e91a/volumes" Jul 06 20:17:09.416414 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.416373 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:09.417503 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.417476 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" containerName="maas-api" Jul 06 20:17:09.417503 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.417503 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" containerName="maas-api" Jul 06 20:17:09.417696 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.417595 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="81488091-e0c8-4a0b-8fc4-6bf28862e91a" containerName="maas-api" Jul 06 20:17:09.422386 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.422366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.424411 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.424389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-d8100888-serving-cert\"" Jul 06 20:17:09.424411 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.424392 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:17:09.430010 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.429981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:09.515842 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.515804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.515842 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.515875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.516154 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.515967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvgz\" (UniqueName: \"kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.616981 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.616942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.617165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.617000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.617165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.617073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvgz\" (UniqueName: \"kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.617758 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.617733 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.619512 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.619488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.625101 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.625078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvgz\" (UniqueName: \"kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz\") pod \"maas-api-e2e-ait-d8100888-996555c48-xkb7n\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.736755 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.736669 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:09.884113 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:09.884077 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:09.887406 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:17:09.887376 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10806176_db51_4784_be8d_66102b98589d.slice/crio-2eac87f53b1629ed8124c7c4b0650cf6448d01cd6460b0e8240f88f7e4eeb787 WatchSource:0}: Error finding container 2eac87f53b1629ed8124c7c4b0650cf6448d01cd6460b0e8240f88f7e4eeb787: Status 404 returned error can't find the container with id 2eac87f53b1629ed8124c7c4b0650cf6448d01cd6460b0e8240f88f7e4eeb787 Jul 06 20:17:10.128617 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:10.128580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" event={"ID":"10806176-db51-4784-be8d-66102b98589d","Type":"ContainerStarted","Data":"2eac87f53b1629ed8124c7c4b0650cf6448d01cd6460b0e8240f88f7e4eeb787"} Jul 06 20:17:11.135528 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:11.135485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" event={"ID":"10806176-db51-4784-be8d-66102b98589d","Type":"ContainerStarted","Data":"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f"} Jul 06 20:17:11.136004 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:11.135611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:11.157102 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:11.157050 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" podStartSLOduration=1.873502631 podStartE2EDuration="2.157032998s" totalImagesPullingTime="283.530367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:17:11.156123147 +0000 UTC m=+1553.997125648" watchObservedRunningTime="2026-07-06 20:17:11.157032998 +0000 UTC m=+1553.998035471" Jul 06 20:17:14.791790 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:14.791749 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:14.792240 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:14.792018 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" podUID="10806176-db51-4784-be8d-66102b98589d" containerName="maas-api" containerID="cri-o://aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f" gracePeriod=30 Jul 06 20:17:15.039697 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.039669 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:15.155024 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.154991 2564 generic.go:356] "Generic (PLEG): container finished" podID="10806176-db51-4784-be8d-66102b98589d" containerID="aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f" exitCode=0 Jul 06 20:17:15.155222 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.155051 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" Jul 06 20:17:15.155222 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.155079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" event={"ID":"10806176-db51-4784-be8d-66102b98589d","Type":"ContainerDied","Data":"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f"} Jul 06 20:17:15.155222 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.155159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n" event={"ID":"10806176-db51-4784-be8d-66102b98589d","Type":"ContainerDied","Data":"2eac87f53b1629ed8124c7c4b0650cf6448d01cd6460b0e8240f88f7e4eeb787"} Jul 06 20:17:15.155222 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.155183 2564 scope.go:117] "RemoveContainer" containerID="aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f" Jul 06 20:17:15.163875 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.163840 2564 scope.go:117] "RemoveContainer" containerID="aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f" Jul 06 20:17:15.164152 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:15.164134 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f\": container with ID starting with aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f not found: ID does not exist" containerID="aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f" Jul 06 20:17:15.164219 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.164160 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f"} err="failed to get container status \"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f\": rpc error: code = NotFound desc = could not find container \"aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f\": container with ID starting with aedd638cf8b2fad30fd8fd2a50ffa92aa561e407ef312d7c3695b62d7c03955f not found: ID does not exist" Jul 06 20:17:15.170518 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.170494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgvgz\" (UniqueName: \"kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz\") pod \"10806176-db51-4784-be8d-66102b98589d\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " Jul 06 20:17:15.170634 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.170584 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls\") pod \"10806176-db51-4784-be8d-66102b98589d\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " Jul 06 20:17:15.170634 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.170618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca\") pod \"10806176-db51-4784-be8d-66102b98589d\" (UID: \"10806176-db51-4784-be8d-66102b98589d\") " Jul 06 20:17:15.171051 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.171027 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10806176-db51-4784-be8d-66102b98589d" (UID: "10806176-db51-4784-be8d-66102b98589d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:15.172593 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.172573 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10806176-db51-4784-be8d-66102b98589d" (UID: "10806176-db51-4784-be8d-66102b98589d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:17:15.172691 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.172673 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz" (OuterVolumeSpecName: "kube-api-access-xgvgz") pod "10806176-db51-4784-be8d-66102b98589d" (UID: "10806176-db51-4784-be8d-66102b98589d"). InnerVolumeSpecName "kube-api-access-xgvgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:17:15.272463 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.272425 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvgz\" (UniqueName: \"kubernetes.io/projected/10806176-db51-4784-be8d-66102b98589d-kube-api-access-xgvgz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:15.272463 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.272455 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10806176-db51-4784-be8d-66102b98589d-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:15.272463 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.272466 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10806176-db51-4784-be8d-66102b98589d-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:15.477971 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.477938 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:15.483195 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.483168 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-d8100888-996555c48-xkb7n"] Jul 06 20:17:15.713038 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:15.713006 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10806176-db51-4784-be8d-66102b98589d" path="/var/lib/kubelet/pods/10806176-db51-4784-be8d-66102b98589d/volumes" Jul 06 20:17:28.440540 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.440502 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:28.441134 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.441118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10806176-db51-4784-be8d-66102b98589d" containerName="maas-api" Jul 06 20:17:28.441195 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.441137 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="10806176-db51-4784-be8d-66102b98589d" containerName="maas-api" Jul 06 20:17:28.441275 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.441264 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="10806176-db51-4784-be8d-66102b98589d" containerName="maas-api" Jul 06 20:17:28.444775 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.444756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.447968 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.447949 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-b341f026-serving-cert\"" Jul 06 20:17:28.448075 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.447999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:17:28.461640 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.461614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:28.505996 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.505950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hdz\" (UniqueName: \"kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.505996 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.505998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.506256 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.506083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.607444 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.607407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hdz\" (UniqueName: \"kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.607444 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.607444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.607731 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.607475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.607731 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:28.607632 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-b341f026-serving-cert: secret "maas-api-e2e-derive-b341f026-serving-cert" not found Jul 06 20:17:28.607874 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:28.607747 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls podName:d3d6ccce-0f08-4bb7-b621-fd8de903e177 nodeName:}" failed. No retries permitted until 2026-07-06 20:17:29.10773041 +0000 UTC m=+1571.948732861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls") pod "maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" (UID: "d3d6ccce-0f08-4bb7-b621-fd8de903e177") : secret "maas-api-e2e-derive-b341f026-serving-cert" not found Jul 06 20:17:28.608204 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.608181 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:28.617139 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:28.617107 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hdz\" (UniqueName: \"kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:29.112509 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:29.112464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:29.112707 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:29.112664 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-b341f026-serving-cert: secret "maas-api-e2e-derive-b341f026-serving-cert" not found Jul 06 20:17:29.112753 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:29.112746 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls podName:d3d6ccce-0f08-4bb7-b621-fd8de903e177 nodeName:}" failed. No retries permitted until 2026-07-06 20:17:30.112724717 +0000 UTC m=+1572.953727181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls") pod "maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" (UID: "d3d6ccce-0f08-4bb7-b621-fd8de903e177") : secret "maas-api-e2e-derive-b341f026-serving-cert" not found Jul 06 20:17:30.123116 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:30.123074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:30.125683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:30.125647 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"maas-api-e2e-derive-b341f026-79cc67b8-vvn2b\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:30.255848 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:30.255805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:30.405007 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:30.404982 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:30.407795 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:17:30.407761 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d6ccce_0f08_4bb7_b621_fd8de903e177.slice/crio-5604dc5a14153b743bf1d218aa648561fb903813122419efca5c10a5159a9fcb WatchSource:0}: Error finding container 5604dc5a14153b743bf1d218aa648561fb903813122419efca5c10a5159a9fcb: Status 404 returned error can't find the container with id 5604dc5a14153b743bf1d218aa648561fb903813122419efca5c10a5159a9fcb Jul 06 20:17:31.236844 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:31.236797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" event={"ID":"d3d6ccce-0f08-4bb7-b621-fd8de903e177","Type":"ContainerStarted","Data":"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931"} Jul 06 20:17:31.237254 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:31.236866 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" event={"ID":"d3d6ccce-0f08-4bb7-b621-fd8de903e177","Type":"ContainerStarted","Data":"5604dc5a14153b743bf1d218aa648561fb903813122419efca5c10a5159a9fcb"} Jul 06 20:17:31.237254 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:31.236895 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:33.054942 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.054852 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" podStartSLOduration=4.61283783 podStartE2EDuration="5.054831085s" totalImagesPullingTime="441.993255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:17:31.268109846 +0000 UTC m=+1574.109112319" watchObservedRunningTime="2026-07-06 20:17:33.054831085 +0000 UTC m=+1575.895833558" Jul 06 20:17:33.056100 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.056072 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:33.245346 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.245308 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" podUID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" containerName="maas-api" containerID="cri-o://e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931" gracePeriod=30 Jul 06 20:17:33.495609 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.495580 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:33.556628 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.556593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55hdz\" (UniqueName: \"kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz\") pod \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " Jul 06 20:17:33.556811 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.556666 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca\") pod \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " Jul 06 20:17:33.556811 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.556694 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") pod \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\" (UID: \"d3d6ccce-0f08-4bb7-b621-fd8de903e177\") " Jul 06 20:17:33.557093 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.557065 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3d6ccce-0f08-4bb7-b621-fd8de903e177" (UID: "d3d6ccce-0f08-4bb7-b621-fd8de903e177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:17:33.558683 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.558661 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz" (OuterVolumeSpecName: "kube-api-access-55hdz") pod "d3d6ccce-0f08-4bb7-b621-fd8de903e177" (UID: "d3d6ccce-0f08-4bb7-b621-fd8de903e177"). InnerVolumeSpecName "kube-api-access-55hdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:17:33.558794 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.558770 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3d6ccce-0f08-4bb7-b621-fd8de903e177" (UID: "d3d6ccce-0f08-4bb7-b621-fd8de903e177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:17:33.657939 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.657812 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55hdz\" (UniqueName: \"kubernetes.io/projected/d3d6ccce-0f08-4bb7-b621-fd8de903e177-kube-api-access-55hdz\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:33.657939 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.657847 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6ccce-0f08-4bb7-b621-fd8de903e177-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:33.657939 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:33.657890 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3d6ccce-0f08-4bb7-b621-fd8de903e177-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:17:34.252358 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.252319 2564 generic.go:356] "Generic (PLEG): container finished" podID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" containerID="e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931" exitCode=0 Jul 06 20:17:34.252358 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.252357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" event={"ID":"d3d6ccce-0f08-4bb7-b621-fd8de903e177","Type":"ContainerDied","Data":"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931"} Jul 06 20:17:34.252883 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.252409 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" Jul 06 20:17:34.252883 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.252436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b" event={"ID":"d3d6ccce-0f08-4bb7-b621-fd8de903e177","Type":"ContainerDied","Data":"5604dc5a14153b743bf1d218aa648561fb903813122419efca5c10a5159a9fcb"} Jul 06 20:17:34.252883 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.252460 2564 scope.go:117] "RemoveContainer" containerID="e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931" Jul 06 20:17:34.264390 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.264354 2564 scope.go:117] "RemoveContainer" containerID="e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931" Jul 06 20:17:34.264822 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:17:34.264795 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931\": container with ID starting with e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931 not found: ID does not exist" containerID="e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931" Jul 06 20:17:34.264948 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.264834 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931"} err="failed to get container status \"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931\": rpc error: code = NotFound desc = could not find container \"e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931\": container with ID starting with e8933f3c9801fcca753f9bfb5cf2c4b9e18b6f821c0bb1d88c914cdd20061931 not found: ID does not exist" Jul 06 20:17:34.273006 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.272978 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:34.276017 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:34.275988 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-b341f026-79cc67b8-vvn2b"] Jul 06 20:17:35.713347 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:17:35.713304 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" path="/var/lib/kubelet/pods/d3d6ccce-0f08-4bb7-b621-fd8de903e177/volumes" Jul 06 20:19:18.014045 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.014001 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:18.014642 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.014588 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" containerName="maas-api" Jul 06 20:19:18.014642 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.014605 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" containerName="maas-api" Jul 06 20:19:18.014756 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.014696 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d6ccce-0f08-4bb7-b621-fd8de903e177" containerName="maas-api" Jul 06 20:19:18.018190 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.018171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.020271 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.020245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-762fdb2a-serving-cert\"" Jul 06 20:19:18.020413 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.020278 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:19:18.030211 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.030182 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:18.095316 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.095280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.095488 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.095354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cgg\" (UniqueName: \"kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.095488 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.095381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.196372 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.196335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.196560 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.196420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cgg\" (UniqueName: \"kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.196560 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.196443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.196560 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:19:18.196557 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-762fdb2a-serving-cert: secret "maas-api-e2e-mt-762fdb2a-serving-cert" not found Jul 06 20:19:18.196674 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:19:18.196613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls podName:90bd5a4f-171d-4faa-b264-c60fd01f8faf nodeName:}" failed. No retries permitted until 2026-07-06 20:19:18.6965975 +0000 UTC m=+1681.537599951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls") pod "maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" (UID: "90bd5a4f-171d-4faa-b264-c60fd01f8faf") : secret "maas-api-e2e-mt-762fdb2a-serving-cert" not found Jul 06 20:19:18.197049 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.197031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.204803 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.204773 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cgg\" (UniqueName: \"kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.702200 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.702161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.704718 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.704687 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") pod \"maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:18.931121 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:18.931085 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:19.073063 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:19.072936 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:19.076055 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:19:19.076023 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bd5a4f_171d_4faa_b264_c60fd01f8faf.slice/crio-82911f3b6e0fe3d75db06150c8334aceff3b4f3d16c6e0e444c705344eb51067 WatchSource:0}: Error finding container 82911f3b6e0fe3d75db06150c8334aceff3b4f3d16c6e0e444c705344eb51067: Status 404 returned error can't find the container with id 82911f3b6e0fe3d75db06150c8334aceff3b4f3d16c6e0e444c705344eb51067 Jul 06 20:19:19.758611 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:19.758514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" event={"ID":"90bd5a4f-171d-4faa-b264-c60fd01f8faf","Type":"ContainerStarted","Data":"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a"} Jul 06 20:19:19.758611 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:19.758565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" event={"ID":"90bd5a4f-171d-4faa-b264-c60fd01f8faf","Type":"ContainerStarted","Data":"82911f3b6e0fe3d75db06150c8334aceff3b4f3d16c6e0e444c705344eb51067"} Jul 06 20:19:19.758837 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:19.758638 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:19.796287 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:19.796234 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" podStartSLOduration=2.406349481 podStartE2EDuration="2.796216601s" totalImagesPullingTime="389.86712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:19:19.795285685 +0000 UTC m=+1682.636288176" watchObservedRunningTime="2026-07-06 20:19:19.796216601 +0000 UTC m=+1682.637219074" Jul 06 20:19:25.769809 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:25.769774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:30.340998 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.340959 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:30.341526 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.341227 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" podUID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" containerName="maas-api" containerID="cri-o://3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a" gracePeriod=30 Jul 06 20:19:30.594091 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.594021 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:30.722256 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.722202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") pod \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " Jul 06 20:19:30.722449 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.722403 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca\") pod \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " Jul 06 20:19:30.722449 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.722437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4cgg\" (UniqueName: \"kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg\") pod \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\" (UID: \"90bd5a4f-171d-4faa-b264-c60fd01f8faf\") " Jul 06 20:19:30.722804 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.722770 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90bd5a4f-171d-4faa-b264-c60fd01f8faf" (UID: "90bd5a4f-171d-4faa-b264-c60fd01f8faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:19:30.724328 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.724300 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90bd5a4f-171d-4faa-b264-c60fd01f8faf" (UID: "90bd5a4f-171d-4faa-b264-c60fd01f8faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:19:30.724518 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.724496 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg" (OuterVolumeSpecName: "kube-api-access-d4cgg") pod "90bd5a4f-171d-4faa-b264-c60fd01f8faf" (UID: "90bd5a4f-171d-4faa-b264-c60fd01f8faf"). InnerVolumeSpecName "kube-api-access-d4cgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:19:30.813961 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.813923 2564 generic.go:356] "Generic (PLEG): container finished" podID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" containerID="3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a" exitCode=0 Jul 06 20:19:30.814158 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.813977 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" Jul 06 20:19:30.814158 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.814012 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" event={"ID":"90bd5a4f-171d-4faa-b264-c60fd01f8faf","Type":"ContainerDied","Data":"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a"} Jul 06 20:19:30.814158 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.814070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n" event={"ID":"90bd5a4f-171d-4faa-b264-c60fd01f8faf","Type":"ContainerDied","Data":"82911f3b6e0fe3d75db06150c8334aceff3b4f3d16c6e0e444c705344eb51067"} Jul 06 20:19:30.814158 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.814094 2564 scope.go:117] "RemoveContainer" containerID="3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a" Jul 06 20:19:30.823398 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.823375 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90bd5a4f-171d-4faa-b264-c60fd01f8faf-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:19:30.823398 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.823403 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90bd5a4f-171d-4faa-b264-c60fd01f8faf-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:19:30.823398 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.823420 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4cgg\" (UniqueName: \"kubernetes.io/projected/90bd5a4f-171d-4faa-b264-c60fd01f8faf-kube-api-access-d4cgg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:19:30.823833 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.823811 2564 scope.go:117] "RemoveContainer" containerID="3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a" Jul 06 20:19:30.824146 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:19:30.824130 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a\": container with ID starting with 3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a not found: ID does not exist" containerID="3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a" Jul 06 20:19:30.824218 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.824156 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a"} err="failed to get container status \"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a\": rpc error: code = NotFound desc = could not find container \"3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a\": container with ID starting with 3a57dd63e4b41c1a8ab9cbb067121164b05fb7312c092a60c57e874070e23c2a not found: ID does not exist" Jul 06 20:19:30.837309 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.837280 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:30.842175 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:30.842151 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-762fdb2a-7bb9887cdb-r2p2n"] Jul 06 20:19:31.712621 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:31.712580 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" path="/var/lib/kubelet/pods/90bd5a4f-171d-4faa-b264-c60fd01f8faf/volumes" Jul 06 20:19:59.320958 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.320925 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:19:59.323423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.321457 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" containerName="maas-api" Jul 06 20:19:59.323423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.321473 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" containerName="maas-api" Jul 06 20:19:59.323423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.321549 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bd5a4f-171d-4faa-b264-c60fd01f8faf" containerName="maas-api" Jul 06 20:19:59.324391 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.324367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.326539 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.326511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a2e2cfb4-serving-cert\"" Jul 06 20:19:59.326824 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.326805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:19:59.336024 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.336000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:19:59.395216 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.395182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.395415 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.395228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2brg\" (UniqueName: \"kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.395415 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.395297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.496383 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.496338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.496580 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.496408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.496580 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.496435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2brg\" (UniqueName: \"kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.496580 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:19:59.496565 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-a2e2cfb4-serving-cert: secret "maas-api-e2e-mt-a2e2cfb4-serving-cert" not found Jul 06 20:19:59.496763 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:19:59.496637 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls podName:6a871e42-a996-44d4-864e-49de3bf70e11 nodeName:}" failed. No retries permitted until 2026-07-06 20:19:59.996620774 +0000 UTC m=+1722.837623226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls") pod "maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" (UID: "6a871e42-a996-44d4-864e-49de3bf70e11") : secret "maas-api-e2e-mt-a2e2cfb4-serving-cert" not found Jul 06 20:19:59.497210 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.497186 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:19:59.505175 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:19:59.505148 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2brg\" (UniqueName: \"kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:00.001487 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.001454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:00.003759 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.003732 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") pod \"maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:00.237945 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.237907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:00.385917 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.385878 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:20:00.389966 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:20:00.389936 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a871e42_a996_44d4_864e_49de3bf70e11.slice/crio-88ed36e70cf8d80f6f94354f01f707ad39eb2abe5f9de79879417f960111b930 WatchSource:0}: Error finding container 88ed36e70cf8d80f6f94354f01f707ad39eb2abe5f9de79879417f960111b930: Status 404 returned error can't find the container with id 88ed36e70cf8d80f6f94354f01f707ad39eb2abe5f9de79879417f960111b930 Jul 06 20:20:00.960390 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.960342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" event={"ID":"6a871e42-a996-44d4-864e-49de3bf70e11","Type":"ContainerStarted","Data":"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0"} Jul 06 20:20:00.960582 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.960442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" event={"ID":"6a871e42-a996-44d4-864e-49de3bf70e11","Type":"ContainerStarted","Data":"88ed36e70cf8d80f6f94354f01f707ad39eb2abe5f9de79879417f960111b930"} Jul 06 20:20:00.960582 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.960554 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:00.978539 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:00.978479 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" podStartSLOduration=1.622238356 podStartE2EDuration="1.97845709s" totalImagesPullingTime="356.218734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:20:00.977770487 +0000 UTC m=+1723.818772983" watchObservedRunningTime="2026-07-06 20:20:00.97845709 +0000 UTC m=+1723.819459563" Jul 06 20:20:06.970455 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:06.970421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:36.601799 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.601761 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:20:36.602481 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.602346 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" podUID="6a871e42-a996-44d4-864e-49de3bf70e11" containerName="maas-api" containerID="cri-o://8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0" gracePeriod=30 Jul 06 20:20:36.861137 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.861112 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:36.968954 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.968916 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2brg\" (UniqueName: \"kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg\") pod \"6a871e42-a996-44d4-864e-49de3bf70e11\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " Jul 06 20:20:36.969150 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.968992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca\") pod \"6a871e42-a996-44d4-864e-49de3bf70e11\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " Jul 06 20:20:36.969150 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.969110 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") pod \"6a871e42-a996-44d4-864e-49de3bf70e11\" (UID: \"6a871e42-a996-44d4-864e-49de3bf70e11\") " Jul 06 20:20:36.969367 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.969337 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a871e42-a996-44d4-864e-49de3bf70e11" (UID: "6a871e42-a996-44d4-864e-49de3bf70e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:20:36.971160 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.971124 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg" (OuterVolumeSpecName: "kube-api-access-c2brg") pod "6a871e42-a996-44d4-864e-49de3bf70e11" (UID: "6a871e42-a996-44d4-864e-49de3bf70e11"). InnerVolumeSpecName "kube-api-access-c2brg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:20:36.971257 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:36.971152 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a871e42-a996-44d4-864e-49de3bf70e11" (UID: "6a871e42-a996-44d4-864e-49de3bf70e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:20:37.070787 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.070745 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a871e42-a996-44d4-864e-49de3bf70e11-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:20:37.070787 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.070777 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2brg\" (UniqueName: \"kubernetes.io/projected/6a871e42-a996-44d4-864e-49de3bf70e11-kube-api-access-c2brg\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:20:37.070787 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.070786 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a871e42-a996-44d4-864e-49de3bf70e11-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:20:37.135208 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.135114 2564 generic.go:356] "Generic (PLEG): container finished" podID="6a871e42-a996-44d4-864e-49de3bf70e11" containerID="8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0" exitCode=0 Jul 06 20:20:37.135208 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.135163 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" Jul 06 20:20:37.135394 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.135199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" event={"ID":"6a871e42-a996-44d4-864e-49de3bf70e11","Type":"ContainerDied","Data":"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0"} Jul 06 20:20:37.135394 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.135248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr" event={"ID":"6a871e42-a996-44d4-864e-49de3bf70e11","Type":"ContainerDied","Data":"88ed36e70cf8d80f6f94354f01f707ad39eb2abe5f9de79879417f960111b930"} Jul 06 20:20:37.135394 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.135279 2564 scope.go:117] "RemoveContainer" containerID="8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0" Jul 06 20:20:37.144717 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.144700 2564 scope.go:117] "RemoveContainer" containerID="8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0" Jul 06 20:20:37.145004 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:20:37.144985 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0\": container with ID starting with 8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0 not found: ID does not exist" containerID="8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0" Jul 06 20:20:37.145085 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.145012 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0"} err="failed to get container status \"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0\": rpc error: code = NotFound desc = could not find container \"8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0\": container with ID starting with 8142d8ee97c721bc93c8eba41eea77e323861c233c317d9a3731b608780ff3e0 not found: ID does not exist" Jul 06 20:20:37.157719 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.157690 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:20:37.162120 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.162096 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a2e2cfb4-75fcc9bcf7-cg4rr"] Jul 06 20:20:37.714242 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:20:37.714209 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a871e42-a996-44d4-864e-49de3bf70e11" path="/var/lib/kubelet/pods/6a871e42-a996-44d4-864e-49de3bf70e11/volumes" Jul 06 20:21:17.786810 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:21:17.786777 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:21:17.792708 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:21:17.792681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:22:31.191401 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.191361 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:22:31.191960 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.191943 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a871e42-a996-44d4-864e-49de3bf70e11" containerName="maas-api" Jul 06 20:22:31.192016 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.191963 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a871e42-a996-44d4-864e-49de3bf70e11" containerName="maas-api" Jul 06 20:22:31.192096 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.192083 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a871e42-a996-44d4-864e-49de3bf70e11" containerName="maas-api" Jul 06 20:22:31.195571 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.195546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.198107 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.198082 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:22:31.198269 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.198127 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-efdc2b-serving-cert\"" Jul 06 20:22:31.212029 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.211999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:22:31.298118 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.298070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.298340 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.298178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79qf\" (UniqueName: \"kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.298340 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.298243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.399757 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.399720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79qf\" (UniqueName: \"kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.400014 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.399774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.400014 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.399850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.400517 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.400491 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.402363 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.402328 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.413714 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.413682 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79qf\" (UniqueName: \"kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf\") pod \"maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.507495 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.507391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:31.662713 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.662674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:22:31.662904 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:22:31.662682 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58f0605_3794_4fac_a7c2_57a34701db9f.slice/crio-41ede43c8b2eb3d70560a64827ab12858a46bd788cf82bde9093af5acb60869c WatchSource:0}: Error finding container 41ede43c8b2eb3d70560a64827ab12858a46bd788cf82bde9093af5acb60869c: Status 404 returned error can't find the container with id 41ede43c8b2eb3d70560a64827ab12858a46bd788cf82bde9093af5acb60869c Jul 06 20:22:31.664118 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.664093 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:22:31.673651 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:31.673617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" event={"ID":"f58f0605-3794-4fac-a7c2-57a34701db9f","Type":"ContainerStarted","Data":"41ede43c8b2eb3d70560a64827ab12858a46bd788cf82bde9093af5acb60869c"} Jul 06 20:22:32.679875 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:32.679823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" event={"ID":"f58f0605-3794-4fac-a7c2-57a34701db9f","Type":"ContainerStarted","Data":"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd"} Jul 06 20:22:32.680429 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:32.680036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:32.743121 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:32.743065 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" podStartSLOduration=1.211007876 podStartE2EDuration="1.743049883s" totalImagesPullingTime="532.042007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:32.741290216 +0000 UTC m=+1875.582292709" watchObservedRunningTime="2026-07-06 20:22:32.743049883 +0000 UTC m=+1875.584052355" Jul 06 20:22:38.689847 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:38.689818 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:22:39.386746 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.386706 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:22:39.390688 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.390670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.396955 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.396923 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-f034b4-serving-cert\"" Jul 06 20:22:39.428833 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.428750 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:22:39.481923 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.481882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5kc\" (UniqueName: \"kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.482128 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.481935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.482128 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.482030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.583267 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.583208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5kc\" (UniqueName: \"kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.583267 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.583271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.583570 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.583309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.583570 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:22:39.583381 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-f034b4-serving-cert: secret "maas-api-e2e-inf-b-f034b4-serving-cert" not found Jul 06 20:22:39.583570 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:22:39.583462 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls podName:191fccb0-28c8-4d9c-a9ff-bcf2e2e37296 nodeName:}" failed. No retries permitted until 2026-07-06 20:22:40.083437878 +0000 UTC m=+1882.924440333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls") pod "maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" (UID: "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296") : secret "maas-api-e2e-inf-b-f034b4-serving-cert" not found Jul 06 20:22:39.583962 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.583944 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:39.624256 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:39.624214 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5kc\" (UniqueName: \"kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:40.088024 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:40.087982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:40.090522 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:40.090493 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") pod \"maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:40.301931 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:40.301884 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:40.471525 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:40.471488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:22:40.474664 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:22:40.474629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191fccb0_28c8_4d9c_a9ff_bcf2e2e37296.slice/crio-9c6d0ef50d188a0b88502a9626a279d225567e6b85673f562e3bf3b288ab5bf2 WatchSource:0}: Error finding container 9c6d0ef50d188a0b88502a9626a279d225567e6b85673f562e3bf3b288ab5bf2: Status 404 returned error can't find the container with id 9c6d0ef50d188a0b88502a9626a279d225567e6b85673f562e3bf3b288ab5bf2 Jul 06 20:22:40.718712 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:40.718676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" event={"ID":"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296","Type":"ContainerStarted","Data":"9c6d0ef50d188a0b88502a9626a279d225567e6b85673f562e3bf3b288ab5bf2"} Jul 06 20:22:41.724587 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:41.724550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" event={"ID":"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296","Type":"ContainerStarted","Data":"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672"} Jul 06 20:22:41.725026 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:41.724617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:22:41.751944 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:41.751846 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" podStartSLOduration=2.361606422 podStartE2EDuration="2.751823412s" totalImagesPullingTime="390.21699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:22:41.751289739 +0000 UTC m=+1884.592292206" watchObservedRunningTime="2026-07-06 20:22:41.751823412 +0000 UTC m=+1884.592825886" Jul 06 20:22:47.735563 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:22:47.735523 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:24:46.028111 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.028074 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:24:46.028614 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.028414 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" podUID="f58f0605-3794-4fac-a7c2-57a34701db9f" containerName="maas-api" containerID="cri-o://69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd" gracePeriod=30 Jul 06 20:24:46.276484 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.276458 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:24:46.326829 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.326736 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls\") pod \"f58f0605-3794-4fac-a7c2-57a34701db9f\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " Jul 06 20:24:46.326829 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.326778 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v79qf\" (UniqueName: \"kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf\") pod \"f58f0605-3794-4fac-a7c2-57a34701db9f\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " Jul 06 20:24:46.326829 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.326823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca\") pod \"f58f0605-3794-4fac-a7c2-57a34701db9f\" (UID: \"f58f0605-3794-4fac-a7c2-57a34701db9f\") " Jul 06 20:24:46.327281 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.327256 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f58f0605-3794-4fac-a7c2-57a34701db9f" (UID: "f58f0605-3794-4fac-a7c2-57a34701db9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:24:46.329101 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.329070 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f58f0605-3794-4fac-a7c2-57a34701db9f" (UID: "f58f0605-3794-4fac-a7c2-57a34701db9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:24:46.329101 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.329082 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf" (OuterVolumeSpecName: "kube-api-access-v79qf") pod "f58f0605-3794-4fac-a7c2-57a34701db9f" (UID: "f58f0605-3794-4fac-a7c2-57a34701db9f"). InnerVolumeSpecName "kube-api-access-v79qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:24:46.346550 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.346523 2564 generic.go:356] "Generic (PLEG): container finished" podID="f58f0605-3794-4fac-a7c2-57a34701db9f" containerID="69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd" exitCode=0 Jul 06 20:24:46.346709 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.346586 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" Jul 06 20:24:46.346709 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.346606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" event={"ID":"f58f0605-3794-4fac-a7c2-57a34701db9f","Type":"ContainerDied","Data":"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd"} Jul 06 20:24:46.346709 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.346662 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz" event={"ID":"f58f0605-3794-4fac-a7c2-57a34701db9f","Type":"ContainerDied","Data":"41ede43c8b2eb3d70560a64827ab12858a46bd788cf82bde9093af5acb60869c"} Jul 06 20:24:46.346709 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.346687 2564 scope.go:117] "RemoveContainer" containerID="69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd" Jul 06 20:24:46.355819 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.355801 2564 scope.go:117] "RemoveContainer" containerID="69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd" Jul 06 20:24:46.356113 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:24:46.356089 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd\": container with ID starting with 69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd not found: ID does not exist" containerID="69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd" Jul 06 20:24:46.356186 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.356117 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd"} err="failed to get container status \"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd\": rpc error: code = NotFound desc = could not find container \"69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd\": container with ID starting with 69f7d35fa042450c5d469f49730f9a9e16f2285e2570d31caf027e732f48e6bd not found: ID does not exist" Jul 06 20:24:46.393593 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.393555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:24:46.408468 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.408437 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-efdc2b-6bff8d8d8b-v9hqz"] Jul 06 20:24:46.428553 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.428518 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f58f0605-3794-4fac-a7c2-57a34701db9f-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:24:46.428553 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.428549 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79qf\" (UniqueName: \"kubernetes.io/projected/f58f0605-3794-4fac-a7c2-57a34701db9f-kube-api-access-v79qf\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:24:46.428737 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:46.428562 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f58f0605-3794-4fac-a7c2-57a34701db9f-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:24:47.715074 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:24:47.715040 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f58f0605-3794-4fac-a7c2-57a34701db9f" path="/var/lib/kubelet/pods/f58f0605-3794-4fac-a7c2-57a34701db9f/volumes" Jul 06 20:25:29.831382 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:29.831337 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:25:29.831946 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:29.831548 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" podUID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" containerName="maas-api" containerID="cri-o://4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672" gracePeriod=30 Jul 06 20:25:30.089720 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.089653 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:25:30.156780 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.156743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca\") pod \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " Jul 06 20:25:30.157018 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.156867 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw5kc\" (UniqueName: \"kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc\") pod \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " Jul 06 20:25:30.157018 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.156983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") pod \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\" (UID: \"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296\") " Jul 06 20:25:30.157233 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.157209 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" (UID: "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:25:30.157438 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.157415 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:25:30.159070 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.159048 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" (UID: "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:25:30.159070 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.159061 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc" (OuterVolumeSpecName: "kube-api-access-fw5kc") pod "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" (UID: "191fccb0-28c8-4d9c-a9ff-bcf2e2e37296"). InnerVolumeSpecName "kube-api-access-fw5kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:25:30.258399 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.258359 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5kc\" (UniqueName: \"kubernetes.io/projected/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-kube-api-access-fw5kc\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:25:30.258399 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.258391 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:25:30.559465 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.559429 2564 generic.go:356] "Generic (PLEG): container finished" podID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" containerID="4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672" exitCode=0 Jul 06 20:25:30.559691 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.559478 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" Jul 06 20:25:30.559691 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.559510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" event={"ID":"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296","Type":"ContainerDied","Data":"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672"} Jul 06 20:25:30.559691 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.559566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx" event={"ID":"191fccb0-28c8-4d9c-a9ff-bcf2e2e37296","Type":"ContainerDied","Data":"9c6d0ef50d188a0b88502a9626a279d225567e6b85673f562e3bf3b288ab5bf2"} Jul 06 20:25:30.559691 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.559589 2564 scope.go:117] "RemoveContainer" containerID="4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672" Jul 06 20:25:30.569160 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.569145 2564 scope.go:117] "RemoveContainer" containerID="4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672" Jul 06 20:25:30.569448 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:25:30.569428 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672\": container with ID starting with 4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672 not found: ID does not exist" containerID="4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672" Jul 06 20:25:30.569522 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.569457 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672"} err="failed to get container status \"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672\": rpc error: code = NotFound desc = could not find container \"4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672\": container with ID starting with 4864d25366e2ed716732dc6e39c101dd00d8ecd4ad69654cd404aaea81d0d672 not found: ID does not exist" Jul 06 20:25:30.605025 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.604989 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:25:30.608901 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:30.608843 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f034b4-d9c9949b8-wqbpx"] Jul 06 20:25:31.712983 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:31.712951 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" path="/var/lib/kubelet/pods/191fccb0-28c8-4d9c-a9ff-bcf2e2e37296/volumes" Jul 06 20:25:40.656460 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656368 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:25:40.656925 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656783 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58f0605-3794-4fac-a7c2-57a34701db9f" containerName="maas-api" Jul 06 20:25:40.656925 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656796 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58f0605-3794-4fac-a7c2-57a34701db9f" containerName="maas-api" Jul 06 20:25:40.656925 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656837 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" containerName="maas-api" Jul 06 20:25:40.656925 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656844 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" containerName="maas-api" Jul 06 20:25:40.657149 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656933 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58f0605-3794-4fac-a7c2-57a34701db9f" containerName="maas-api" Jul 06 20:25:40.657149 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.656946 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="191fccb0-28c8-4d9c-a9ff-bcf2e2e37296" containerName="maas-api" Jul 06 20:25:40.661246 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.661227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.663833 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.663813 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:25:40.664196 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.664171 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-75489c-serving-cert\"" Jul 06 20:25:40.678261 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.678232 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:25:40.760393 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.760359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.760393 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.760390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqss\" (UniqueName: \"kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.760676 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.760529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.861444 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.861400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.861444 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.861444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqss\" (UniqueName: \"kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.861720 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.861514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.862107 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.862083 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.864039 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.864018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.873933 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.873908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqss\" (UniqueName: \"kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss\") pod \"maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:40.971469 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:40.971367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:41.135264 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:41.135239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:25:41.137633 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:25:41.137602 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce65a736_41cd_49d4_9718_9157d6400b57.slice/crio-aa47f7f7ac533a43b24c3ab620ebde0abb8727bdbaaf206c18233f97bd4a83f9 WatchSource:0}: Error finding container aa47f7f7ac533a43b24c3ab620ebde0abb8727bdbaaf206c18233f97bd4a83f9: Status 404 returned error can't find the container with id aa47f7f7ac533a43b24c3ab620ebde0abb8727bdbaaf206c18233f97bd4a83f9 Jul 06 20:25:41.616895 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:41.616834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" event={"ID":"ce65a736-41cd-49d4-9718-9157d6400b57","Type":"ContainerStarted","Data":"aa47f7f7ac533a43b24c3ab620ebde0abb8727bdbaaf206c18233f97bd4a83f9"} Jul 06 20:25:42.623375 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:42.623339 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" event={"ID":"ce65a736-41cd-49d4-9718-9157d6400b57","Type":"ContainerStarted","Data":"f7be89ac2e307cfca697b63082d530bd32f89315b57ca06bcb005fb898e4f6e4"} Jul 06 20:25:42.623874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:42.623532 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:42.651083 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:42.651027 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" podStartSLOduration=2.096915297 podStartE2EDuration="2.651008876s" totalImagesPullingTime="554.093579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:42.648607101 +0000 UTC m=+2065.489609611" watchObservedRunningTime="2026-07-06 20:25:42.651008876 +0000 UTC m=+2065.492011350" Jul 06 20:25:48.521984 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.521949 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:25:48.525442 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.525424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.529056 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.529031 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-3057cf-serving-cert\"" Jul 06 20:25:48.559305 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.559275 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:25:48.635874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.635828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:25:48.636751 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.636725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.636909 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.636769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrdn\" (UniqueName: \"kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.636909 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.636847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.738280 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.738239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.738487 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.738308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrdn\" (UniqueName: \"kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.738571 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.738501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.738666 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:25:48.738644 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-3057cf-serving-cert: secret "maas-api-e2e-api-b-3057cf-serving-cert" not found Jul 06 20:25:48.738761 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:25:48.738747 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls podName:09d00c8f-492c-4f6d-8fce-9527848866c3 nodeName:}" failed. No retries permitted until 2026-07-06 20:25:49.2387253 +0000 UTC m=+2072.079727769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls") pod "maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" (UID: "09d00c8f-492c-4f6d-8fce-9527848866c3") : secret "maas-api-e2e-api-b-3057cf-serving-cert" not found Jul 06 20:25:48.739082 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.739064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:48.765260 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:48.765228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrdn\" (UniqueName: \"kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:49.244345 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:49.244301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:49.246988 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:49.246959 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") pod \"maas-api-e2e-api-b-3057cf-85f796f85b-szh6r\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:49.437157 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:49.437119 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:49.584772 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:49.584737 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:25:49.588824 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:25:49.588789 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d00c8f_492c_4f6d_8fce_9527848866c3.slice/crio-3e6ef90663b3e4994afc3602eba25a002f0ef5bca6db574de941b856d679e306 WatchSource:0}: Error finding container 3e6ef90663b3e4994afc3602eba25a002f0ef5bca6db574de941b856d679e306: Status 404 returned error can't find the container with id 3e6ef90663b3e4994afc3602eba25a002f0ef5bca6db574de941b856d679e306 Jul 06 20:25:49.667040 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:49.666998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" event={"ID":"09d00c8f-492c-4f6d-8fce-9527848866c3","Type":"ContainerStarted","Data":"3e6ef90663b3e4994afc3602eba25a002f0ef5bca6db574de941b856d679e306"} Jul 06 20:25:50.673191 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:50.673154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" event={"ID":"09d00c8f-492c-4f6d-8fce-9527848866c3","Type":"ContainerStarted","Data":"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d"} Jul 06 20:25:50.673645 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:50.673343 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:25:50.713982 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:50.713930 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" podStartSLOduration=2.393904613 podStartE2EDuration="2.713913872s" totalImagesPullingTime="320.009259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:25:50.712034742 +0000 UTC m=+2073.553037215" watchObservedRunningTime="2026-07-06 20:25:50.713913872 +0000 UTC m=+2073.554916345" Jul 06 20:25:56.683021 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:25:56.682990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:26:02.601170 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.601137 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:26:02.601595 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.601393 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" podUID="ce65a736-41cd-49d4-9718-9157d6400b57" containerName="maas-api" containerID="cri-o://f7be89ac2e307cfca697b63082d530bd32f89315b57ca06bcb005fb898e4f6e4" gracePeriod=30 Jul 06 20:26:02.735141 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.735104 2564 generic.go:356] "Generic (PLEG): container finished" podID="ce65a736-41cd-49d4-9718-9157d6400b57" containerID="f7be89ac2e307cfca697b63082d530bd32f89315b57ca06bcb005fb898e4f6e4" exitCode=0 Jul 06 20:26:02.735329 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.735180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" event={"ID":"ce65a736-41cd-49d4-9718-9157d6400b57","Type":"ContainerDied","Data":"f7be89ac2e307cfca697b63082d530bd32f89315b57ca06bcb005fb898e4f6e4"} Jul 06 20:26:02.857740 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.857719 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:26:02.878795 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.878759 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrqss\" (UniqueName: \"kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss\") pod \"ce65a736-41cd-49d4-9718-9157d6400b57\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " Jul 06 20:26:02.878990 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.878883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca\") pod \"ce65a736-41cd-49d4-9718-9157d6400b57\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " Jul 06 20:26:02.878990 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.878967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls\") pod \"ce65a736-41cd-49d4-9718-9157d6400b57\" (UID: \"ce65a736-41cd-49d4-9718-9157d6400b57\") " Jul 06 20:26:02.879324 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.879291 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce65a736-41cd-49d4-9718-9157d6400b57" (UID: "ce65a736-41cd-49d4-9718-9157d6400b57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:26:02.881168 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.881141 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce65a736-41cd-49d4-9718-9157d6400b57" (UID: "ce65a736-41cd-49d4-9718-9157d6400b57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:26:02.881168 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.881154 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss" (OuterVolumeSpecName: "kube-api-access-nrqss") pod "ce65a736-41cd-49d4-9718-9157d6400b57" (UID: "ce65a736-41cd-49d4-9718-9157d6400b57"). InnerVolumeSpecName "kube-api-access-nrqss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:26:02.979892 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.979817 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce65a736-41cd-49d4-9718-9157d6400b57-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:02.979892 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.979850 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqss\" (UniqueName: \"kubernetes.io/projected/ce65a736-41cd-49d4-9718-9157d6400b57-kube-api-access-nrqss\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:02.979892 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:02.979891 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce65a736-41cd-49d4-9718-9157d6400b57-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:03.741584 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:03.741554 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" Jul 06 20:26:03.742039 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:03.741553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc" event={"ID":"ce65a736-41cd-49d4-9718-9157d6400b57","Type":"ContainerDied","Data":"aa47f7f7ac533a43b24c3ab620ebde0abb8727bdbaaf206c18233f97bd4a83f9"} Jul 06 20:26:03.742039 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:03.741690 2564 scope.go:117] "RemoveContainer" containerID="f7be89ac2e307cfca697b63082d530bd32f89315b57ca06bcb005fb898e4f6e4" Jul 06 20:26:03.765061 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:03.765030 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:26:03.769872 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:03.769836 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-75489c-7998b9bcf9-sdxqc"] Jul 06 20:26:05.712533 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:05.712491 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce65a736-41cd-49d4-9718-9157d6400b57" path="/var/lib/kubelet/pods/ce65a736-41cd-49d4-9718-9157d6400b57/volumes" Jul 06 20:26:10.254121 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.254089 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:26:10.254517 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.254351 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" podUID="09d00c8f-492c-4f6d-8fce-9527848866c3" containerName="maas-api" containerID="cri-o://c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d" gracePeriod=30 Jul 06 20:26:10.510382 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.510317 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:26:10.554974 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.554943 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") pod \"09d00c8f-492c-4f6d-8fce-9527848866c3\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " Jul 06 20:26:10.555153 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.555005 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcrdn\" (UniqueName: \"kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn\") pod \"09d00c8f-492c-4f6d-8fce-9527848866c3\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " Jul 06 20:26:10.555153 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.555122 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca\") pod \"09d00c8f-492c-4f6d-8fce-9527848866c3\" (UID: \"09d00c8f-492c-4f6d-8fce-9527848866c3\") " Jul 06 20:26:10.555468 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.555442 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09d00c8f-492c-4f6d-8fce-9527848866c3" (UID: "09d00c8f-492c-4f6d-8fce-9527848866c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:26:10.557190 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.557161 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09d00c8f-492c-4f6d-8fce-9527848866c3" (UID: "09d00c8f-492c-4f6d-8fce-9527848866c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:26:10.557283 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.557219 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn" (OuterVolumeSpecName: "kube-api-access-tcrdn") pod "09d00c8f-492c-4f6d-8fce-9527848866c3" (UID: "09d00c8f-492c-4f6d-8fce-9527848866c3"). InnerVolumeSpecName "kube-api-access-tcrdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:26:10.656791 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.656756 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d00c8f-492c-4f6d-8fce-9527848866c3-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:10.656791 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.656784 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d00c8f-492c-4f6d-8fce-9527848866c3-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:10.656791 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.656793 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcrdn\" (UniqueName: \"kubernetes.io/projected/09d00c8f-492c-4f6d-8fce-9527848866c3-kube-api-access-tcrdn\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:26:10.778835 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.778741 2564 generic.go:356] "Generic (PLEG): container finished" podID="09d00c8f-492c-4f6d-8fce-9527848866c3" containerID="c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d" exitCode=0 Jul 06 20:26:10.778835 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.778791 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" Jul 06 20:26:10.778835 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.778796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" event={"ID":"09d00c8f-492c-4f6d-8fce-9527848866c3","Type":"ContainerDied","Data":"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d"} Jul 06 20:26:10.778835 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.778838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r" event={"ID":"09d00c8f-492c-4f6d-8fce-9527848866c3","Type":"ContainerDied","Data":"3e6ef90663b3e4994afc3602eba25a002f0ef5bca6db574de941b856d679e306"} Jul 06 20:26:10.779201 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.778877 2564 scope.go:117] "RemoveContainer" containerID="c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d" Jul 06 20:26:10.788244 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.788224 2564 scope.go:117] "RemoveContainer" containerID="c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d" Jul 06 20:26:10.788492 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:26:10.788476 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d\": container with ID starting with c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d not found: ID does not exist" containerID="c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d" Jul 06 20:26:10.788540 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.788501 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d"} err="failed to get container status \"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d\": rpc error: code = NotFound desc = could not find container \"c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d\": container with ID starting with c2060ac18df8936dde72d9ea3ccb606a53d64028a25c305e82d44a042745551d not found: ID does not exist" Jul 06 20:26:10.803690 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.803661 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:26:10.809789 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:10.809763 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-3057cf-85f796f85b-szh6r"] Jul 06 20:26:11.713382 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:11.713346 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d00c8f-492c-4f6d-8fce-9527848866c3" path="/var/lib/kubelet/pods/09d00c8f-492c-4f6d-8fce-9527848866c3/volumes" Jul 06 20:26:17.818977 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:17.818944 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:26:17.828624 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:17.828598 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:26:20.789929 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.789888 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790343 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d00c8f-492c-4f6d-8fce-9527848866c3" containerName="maas-api" Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790361 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d00c8f-492c-4f6d-8fce-9527848866c3" containerName="maas-api" Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790391 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce65a736-41cd-49d4-9718-9157d6400b57" containerName="maas-api" Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790401 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce65a736-41cd-49d4-9718-9157d6400b57" containerName="maas-api" Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790532 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d00c8f-492c-4f6d-8fce-9527848866c3" containerName="maas-api" Jul 06 20:26:20.790574 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.790545 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce65a736-41cd-49d4-9718-9157d6400b57" containerName="maas-api" Jul 06 20:26:20.795163 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.795140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:20.798874 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.798840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tlfs2\"" Jul 06 20:26:20.799252 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.799233 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-629470-serving-cert\"" Jul 06 20:26:20.818414 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.818384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:26:20.966676 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.966631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:20.966938 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.966704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:20.966938 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:20.966807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qq8\" (UniqueName: \"kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.067724 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.067622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.067724 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.067677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.068028 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.067752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qq8\" (UniqueName: \"kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.068416 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.068391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.070251 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.070228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.080843 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.080821 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qq8\" (UniqueName: \"kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8\") pod \"maas-api-e2e-shared-a-629470-7754694dcb-cgjpc\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.106871 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.106834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.286334 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.286299 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:26:21.289911 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:26:21.289875 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9764620_e05a_43ed_bb91_d55b5c718b1f.slice/crio-82d8c2a5de8990f194bbe167c4c814c444706464b6465da8c7707a89b43847d2 WatchSource:0}: Error finding container 82d8c2a5de8990f194bbe167c4c814c444706464b6465da8c7707a89b43847d2: Status 404 returned error can't find the container with id 82d8c2a5de8990f194bbe167c4c814c444706464b6465da8c7707a89b43847d2 Jul 06 20:26:21.833988 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.833950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" event={"ID":"c9764620-e05a-43ed-bb91-d55b5c718b1f","Type":"ContainerStarted","Data":"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f"} Jul 06 20:26:21.834423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.834013 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:21.834423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.834030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" event={"ID":"c9764620-e05a-43ed-bb91-d55b5c718b1f","Type":"ContainerStarted","Data":"82d8c2a5de8990f194bbe167c4c814c444706464b6465da8c7707a89b43847d2"} Jul 06 20:26:21.854676 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:21.854618 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" podStartSLOduration=1.495430235 podStartE2EDuration="1.854600829s" totalImagesPullingTime="359.170594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:21.854487267 +0000 UTC m=+2104.695489750" watchObservedRunningTime="2026-07-06 20:26:21.854600829 +0000 UTC m=+2104.695603303" Jul 06 20:26:27.844087 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:27.844059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:26:28.865611 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.865577 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:26:28.869334 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.869315 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:28.875281 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.875249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-a90d79-serving-cert\"" Jul 06 20:26:28.896986 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.896953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:26:28.939005 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.938972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:28.939192 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.939018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:28.939192 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:28.939096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psr9d\" (UniqueName: \"kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.039636 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.039593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psr9d\" (UniqueName: \"kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.039825 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.039724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.039825 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.039771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.039951 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:26:29.039902 2564 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-a90d79-serving-cert: secret "maas-api-e2e-shared-b-a90d79-serving-cert" not found Jul 06 20:26:29.039993 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:26:29.039970 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls podName:84f00f03-4d58-43d4-93c1-551d4257d2ea nodeName:}" failed. No retries permitted until 2026-07-06 20:26:29.53995086 +0000 UTC m=+2112.380953316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls") pod "maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" (UID: "84f00f03-4d58-43d4-93c1-551d4257d2ea") : secret "maas-api-e2e-shared-b-a90d79-serving-cert" not found Jul 06 20:26:29.040456 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.040437 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.067244 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.067213 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psr9d\" (UniqueName: \"kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.543413 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.543372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.545903 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.545853 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") pod \"maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.780639 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.780594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:29.943348 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:29.943322 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:26:29.946254 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:26:29.946224 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f00f03_4d58_43d4_93c1_551d4257d2ea.slice/crio-df29d01eb653eaee693bf3b5b6ed66d553f81732b4791fbb5a8442b02d72a828 WatchSource:0}: Error finding container df29d01eb653eaee693bf3b5b6ed66d553f81732b4791fbb5a8442b02d72a828: Status 404 returned error can't find the container with id df29d01eb653eaee693bf3b5b6ed66d553f81732b4791fbb5a8442b02d72a828 Jul 06 20:26:30.879460 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:30.879420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" event={"ID":"84f00f03-4d58-43d4-93c1-551d4257d2ea","Type":"ContainerStarted","Data":"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545"} Jul 06 20:26:30.879659 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:30.879506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" event={"ID":"84f00f03-4d58-43d4-93c1-551d4257d2ea","Type":"ContainerStarted","Data":"df29d01eb653eaee693bf3b5b6ed66d553f81732b4791fbb5a8442b02d72a828"} Jul 06 20:26:30.879659 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:30.879559 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:26:30.900622 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:30.900569 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" podStartSLOduration=2.544570409 podStartE2EDuration="2.900550632s" totalImagesPullingTime="355.980223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:26:30.899205209 +0000 UTC m=+2113.740207700" watchObservedRunningTime="2026-07-06 20:26:30.900550632 +0000 UTC m=+2113.741553105" Jul 06 20:26:36.889916 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:26:36.889886 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:30:00.182159 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.182114 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt"] Jul 06 20:30:00.185893 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.185845 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.188182 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:30:00.188147 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.188445 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.188429 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg"] Jul 06 20:30:00.191538 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.191519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:00.193361 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:30:00.193333 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt\" is forbidden: User \"system:node:ip-10-0-140-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-110.ec2.internal' and this object" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.194307 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.194288 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-mkqdh"] Jul 06 20:30:00.197448 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.197435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:00.217188 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.217166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt"] Jul 06 20:30:00.225879 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.225837 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg"] Jul 06 20:30:00.229950 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.229930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-mkqdh"] Jul 06 20:30:00.303558 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.303535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7gt\" (UniqueName: \"kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt\") pod \"maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt\" (UID: \"87b0b74a-17ee-45e2-a3a2-22661fbff7fd\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.303689 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.303637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxl7\" (UniqueName: \"kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7\") pod \"maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg\" (UID: \"7124bba1-8248-43d0-b6d7-51c63afe1a1c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:00.303689 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.303663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xq8\" (UniqueName: \"kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8\") pod \"maas-api-key-cleanup-29722830-mkqdh\" (UID: \"edd1039d-140c-4bb3-a9a1-dc28775229b8\") " pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:00.404715 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.404687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxl7\" (UniqueName: \"kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7\") pod \"maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg\" (UID: \"7124bba1-8248-43d0-b6d7-51c63afe1a1c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:00.404833 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.404727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xq8\" (UniqueName: \"kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8\") pod \"maas-api-key-cleanup-29722830-mkqdh\" (UID: \"edd1039d-140c-4bb3-a9a1-dc28775229b8\") " pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:00.404833 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.404769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7gt\" (UniqueName: \"kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt\") pod \"maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt\" (UID: \"87b0b74a-17ee-45e2-a3a2-22661fbff7fd\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.425875 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.425830 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xq8\" (UniqueName: \"kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8\") pod \"maas-api-key-cleanup-29722830-mkqdh\" (UID: \"edd1039d-140c-4bb3-a9a1-dc28775229b8\") " pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:00.428753 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.428725 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxl7\" (UniqueName: \"kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7\") pod \"maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg\" (UID: \"7124bba1-8248-43d0-b6d7-51c63afe1a1c\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:00.432553 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.432490 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7gt\" (UniqueName: \"kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt\") pod \"maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt\" (UID: \"87b0b74a-17ee-45e2-a3a2-22661fbff7fd\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.495933 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.495915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:00.504441 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.504424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:00.510324 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.510306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:00.664127 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.664085 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt"] Jul 06 20:30:00.668214 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:30:00.668101 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b0b74a_17ee_45e2_a3a2_22661fbff7fd.slice/crio-26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68 WatchSource:0}: Error finding container 26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68: Status 404 returned error can't find the container with id 26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68 Jul 06 20:30:00.705294 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.705270 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722830-mkqdh"] Jul 06 20:30:00.706334 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:30:00.706308 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd1039d_140c_4bb3_a9a1_dc28775229b8.slice/crio-8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7 WatchSource:0}: Error finding container 8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7: Status 404 returned error can't find the container with id 8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7 Jul 06 20:30:00.887012 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.886975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" event={"ID":"edd1039d-140c-4bb3-a9a1-dc28775229b8","Type":"ContainerStarted","Data":"8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7"} Jul 06 20:30:00.888076 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.888053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" event={"ID":"87b0b74a-17ee-45e2-a3a2-22661fbff7fd","Type":"ContainerStarted","Data":"26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68"} Jul 06 20:30:00.892272 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:00.892251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg"] Jul 06 20:30:00.898493 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:30:00.898470 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7124bba1_8248_43d0_b6d7_51c63afe1a1c.slice/crio-f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188 WatchSource:0}: Error finding container f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188: Status 404 returned error can't find the container with id f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188 Jul 06 20:30:01.893523 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.893492 2564 generic.go:356] "Generic (PLEG): container finished" podID="7124bba1-8248-43d0-b6d7-51c63afe1a1c" containerID="5b3a3472b1ff3b1e06540bf71698c891e39bc217f4e43c47b0ae7972a730dd00" exitCode=0 Jul 06 20:30:01.893899 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.893579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" event={"ID":"7124bba1-8248-43d0-b6d7-51c63afe1a1c","Type":"ContainerDied","Data":"5b3a3472b1ff3b1e06540bf71698c891e39bc217f4e43c47b0ae7972a730dd00"} Jul 06 20:30:01.893899 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.893633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" event={"ID":"7124bba1-8248-43d0-b6d7-51c63afe1a1c","Type":"ContainerStarted","Data":"f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188"} Jul 06 20:30:01.895019 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.894978 2564 generic.go:356] "Generic (PLEG): container finished" podID="edd1039d-140c-4bb3-a9a1-dc28775229b8" containerID="c0d86cf2ba529d796bc89eb9e877351b250e34283d442e549f9648355a13502f" exitCode=0 Jul 06 20:30:01.895144 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.895042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" event={"ID":"edd1039d-140c-4bb3-a9a1-dc28775229b8","Type":"ContainerDied","Data":"c0d86cf2ba529d796bc89eb9e877351b250e34283d442e549f9648355a13502f"} Jul 06 20:30:01.896464 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.896444 2564 generic.go:356] "Generic (PLEG): container finished" podID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" containerID="7f52b3a09edbf3b3d66ed9574597c702627c1d3f30a43df987b949efdfcc777b" exitCode=0 Jul 06 20:30:01.896558 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:01.896518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" event={"ID":"87b0b74a-17ee-45e2-a3a2-22661fbff7fd","Type":"ContainerDied","Data":"7f52b3a09edbf3b3d66ed9574597c702627c1d3f30a43df987b949efdfcc777b"} Jul 06 20:30:03.062645 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.062622 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:03.065320 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.065300 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:03.091212 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.091194 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:03.129356 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.129330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs7gt\" (UniqueName: \"kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt\") pod \"87b0b74a-17ee-45e2-a3a2-22661fbff7fd\" (UID: \"87b0b74a-17ee-45e2-a3a2-22661fbff7fd\") " Jul 06 20:30:03.129511 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.129362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xq8\" (UniqueName: \"kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8\") pod \"edd1039d-140c-4bb3-a9a1-dc28775229b8\" (UID: \"edd1039d-140c-4bb3-a9a1-dc28775229b8\") " Jul 06 20:30:03.131516 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.131483 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8" (OuterVolumeSpecName: "kube-api-access-c4xq8") pod "edd1039d-140c-4bb3-a9a1-dc28775229b8" (UID: "edd1039d-140c-4bb3-a9a1-dc28775229b8"). InnerVolumeSpecName "kube-api-access-c4xq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:03.131636 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.131515 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt" (OuterVolumeSpecName: "kube-api-access-fs7gt") pod "87b0b74a-17ee-45e2-a3a2-22661fbff7fd" (UID: "87b0b74a-17ee-45e2-a3a2-22661fbff7fd"). InnerVolumeSpecName "kube-api-access-fs7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:03.230561 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.230477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbxl7\" (UniqueName: \"kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7\") pod \"7124bba1-8248-43d0-b6d7-51c63afe1a1c\" (UID: \"7124bba1-8248-43d0-b6d7-51c63afe1a1c\") " Jul 06 20:30:03.230766 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.230752 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7gt\" (UniqueName: \"kubernetes.io/projected/87b0b74a-17ee-45e2-a3a2-22661fbff7fd-kube-api-access-fs7gt\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:30:03.230817 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.230768 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xq8\" (UniqueName: \"kubernetes.io/projected/edd1039d-140c-4bb3-a9a1-dc28775229b8-kube-api-access-c4xq8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:30:03.232588 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.232564 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7" (OuterVolumeSpecName: "kube-api-access-vbxl7") pod "7124bba1-8248-43d0-b6d7-51c63afe1a1c" (UID: "7124bba1-8248-43d0-b6d7-51c63afe1a1c"). InnerVolumeSpecName "kube-api-access-vbxl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:30:03.331502 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.331470 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbxl7\" (UniqueName: \"kubernetes.io/projected/7124bba1-8248-43d0-b6d7-51c63afe1a1c-kube-api-access-vbxl7\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:30:03.906398 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.906372 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" Jul 06 20:30:03.906398 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.906392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg" event={"ID":"7124bba1-8248-43d0-b6d7-51c63afe1a1c","Type":"ContainerDied","Data":"f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188"} Jul 06 20:30:03.906641 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.906438 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f743d17f3744554af9a684f9f0cb99c94614470ff6f537731586467af0733188" Jul 06 20:30:03.907711 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.907693 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" Jul 06 20:30:03.907836 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.907708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722830-mkqdh" event={"ID":"edd1039d-140c-4bb3-a9a1-dc28775229b8","Type":"ContainerDied","Data":"8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7"} Jul 06 20:30:03.907836 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.907744 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8907424ba898e34bba57bed89794720e295e48c48da1763aff2f9dcd020de7b7" Jul 06 20:30:03.909445 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.909421 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" Jul 06 20:30:03.909566 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.909471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt" event={"ID":"87b0b74a-17ee-45e2-a3a2-22661fbff7fd","Type":"ContainerDied","Data":"26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68"} Jul 06 20:30:03.909566 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:30:03.909506 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a1fe4ea2a641decf216bb1dddb5e165bdca7b2b9edbd385d880ad6991a6c68" Jul 06 20:31:17.852514 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:31:17.852490 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:31:17.862954 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:31:17.862925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qb4xh_f57894ae-5abb-42ec-b9d1-5e4cd912760d/ovn-acl-logging/0.log" Jul 06 20:32:45.719389 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.719351 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:32:45.720119 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.719580 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" podUID="c9764620-e05a-43ed-bb91-d55b5c718b1f" containerName="maas-api" containerID="cri-o://29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f" gracePeriod=30 Jul 06 20:32:45.756029 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.756006 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt"] Jul 06 20:32:45.762252 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.762233 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-629470-29722830-gkxxt"] Jul 06 20:32:45.967942 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.967918 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:32:45.984705 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.984662 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2qq8\" (UniqueName: \"kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8\") pod \"c9764620-e05a-43ed-bb91-d55b5c718b1f\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " Jul 06 20:32:45.984705 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.984701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls\") pod \"c9764620-e05a-43ed-bb91-d55b5c718b1f\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " Jul 06 20:32:45.984823 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.984723 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca\") pod \"c9764620-e05a-43ed-bb91-d55b5c718b1f\" (UID: \"c9764620-e05a-43ed-bb91-d55b5c718b1f\") " Jul 06 20:32:45.985129 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.985109 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9764620-e05a-43ed-bb91-d55b5c718b1f" (UID: "c9764620-e05a-43ed-bb91-d55b5c718b1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:32:45.987811 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.986944 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8" (OuterVolumeSpecName: "kube-api-access-d2qq8") pod "c9764620-e05a-43ed-bb91-d55b5c718b1f" (UID: "c9764620-e05a-43ed-bb91-d55b5c718b1f"). InnerVolumeSpecName "kube-api-access-d2qq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:32:45.988355 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:45.988323 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9764620-e05a-43ed-bb91-d55b5c718b1f" (UID: "c9764620-e05a-43ed-bb91-d55b5c718b1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:32:46.086255 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.086229 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qq8\" (UniqueName: \"kubernetes.io/projected/c9764620-e05a-43ed-bb91-d55b5c718b1f-kube-api-access-d2qq8\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:32:46.086255 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.086250 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9764620-e05a-43ed-bb91-d55b5c718b1f-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:32:46.086255 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.086260 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9764620-e05a-43ed-bb91-d55b5c718b1f-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:32:46.691111 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.691077 2564 generic.go:356] "Generic (PLEG): container finished" podID="c9764620-e05a-43ed-bb91-d55b5c718b1f" containerID="29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f" exitCode=0 Jul 06 20:32:46.691326 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.691127 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" Jul 06 20:32:46.691326 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.691170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" event={"ID":"c9764620-e05a-43ed-bb91-d55b5c718b1f","Type":"ContainerDied","Data":"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f"} Jul 06 20:32:46.691326 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.691236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc" event={"ID":"c9764620-e05a-43ed-bb91-d55b5c718b1f","Type":"ContainerDied","Data":"82d8c2a5de8990f194bbe167c4c814c444706464b6465da8c7707a89b43847d2"} Jul 06 20:32:46.691326 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.691266 2564 scope.go:117] "RemoveContainer" containerID="29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f" Jul 06 20:32:46.700380 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.700367 2564 scope.go:117] "RemoveContainer" containerID="29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f" Jul 06 20:32:46.700621 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:32:46.700599 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f\": container with ID starting with 29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f not found: ID does not exist" containerID="29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f" Jul 06 20:32:46.700689 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.700627 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f"} err="failed to get container status \"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f\": rpc error: code = NotFound desc = could not find container \"29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f\": container with ID starting with 29e97cb4586be95e36e5ea1db73ae5b354e92e9ec0628a2f5fcd06dfbbf03a2f not found: ID does not exist" Jul 06 20:32:46.715821 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.715799 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:32:46.717511 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:46.717492 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-629470-7754694dcb-cgjpc"] Jul 06 20:32:47.713090 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:47.713057 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" path="/var/lib/kubelet/pods/87b0b74a-17ee-45e2-a3a2-22661fbff7fd/volumes" Jul 06 20:32:47.713606 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:32:47.713584 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9764620-e05a-43ed-bb91-d55b5c718b1f" path="/var/lib/kubelet/pods/c9764620-e05a-43ed-bb91-d55b5c718b1f/volumes" Jul 06 20:33:16.088252 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.088174 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:33:16.088677 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.088392 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" podUID="84f00f03-4d58-43d4-93c1-551d4257d2ea" containerName="maas-api" containerID="cri-o://0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545" gracePeriod=30 Jul 06 20:33:16.139406 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.139384 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg"] Jul 06 20:33:16.145991 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.145971 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-a90d79-29722830-knxvg"] Jul 06 20:33:16.335684 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.335662 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:33:16.444595 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.444573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca\") pod \"84f00f03-4d58-43d4-93c1-551d4257d2ea\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " Jul 06 20:33:16.444762 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.444601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") pod \"84f00f03-4d58-43d4-93c1-551d4257d2ea\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " Jul 06 20:33:16.444762 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.444716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psr9d\" (UniqueName: \"kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d\") pod \"84f00f03-4d58-43d4-93c1-551d4257d2ea\" (UID: \"84f00f03-4d58-43d4-93c1-551d4257d2ea\") " Jul 06 20:33:16.444998 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.444967 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84f00f03-4d58-43d4-93c1-551d4257d2ea" (UID: "84f00f03-4d58-43d4-93c1-551d4257d2ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 20:33:16.446625 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.446595 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84f00f03-4d58-43d4-93c1-551d4257d2ea" (UID: "84f00f03-4d58-43d4-93c1-551d4257d2ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 20:33:16.446726 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.446672 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d" (OuterVolumeSpecName: "kube-api-access-psr9d") pod "84f00f03-4d58-43d4-93c1-551d4257d2ea" (UID: "84f00f03-4d58-43d4-93c1-551d4257d2ea"). InnerVolumeSpecName "kube-api-access-psr9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 20:33:16.545580 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.545543 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psr9d\" (UniqueName: \"kubernetes.io/projected/84f00f03-4d58-43d4-93c1-551d4257d2ea-kube-api-access-psr9d\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:33:16.545580 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.545573 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f00f03-4d58-43d4-93c1-551d4257d2ea-openshift-service-ca\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:33:16.545580 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.545582 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84f00f03-4d58-43d4-93c1-551d4257d2ea-maas-api-tls\") on node \"ip-10-0-140-110.ec2.internal\" DevicePath \"\"" Jul 06 20:33:16.836804 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.836720 2564 generic.go:356] "Generic (PLEG): container finished" podID="84f00f03-4d58-43d4-93c1-551d4257d2ea" containerID="0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545" exitCode=0 Jul 06 20:33:16.836804 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.836767 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" Jul 06 20:33:16.837010 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.836804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" event={"ID":"84f00f03-4d58-43d4-93c1-551d4257d2ea","Type":"ContainerDied","Data":"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545"} Jul 06 20:33:16.837010 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.836853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg" event={"ID":"84f00f03-4d58-43d4-93c1-551d4257d2ea","Type":"ContainerDied","Data":"df29d01eb653eaee693bf3b5b6ed66d553f81732b4791fbb5a8442b02d72a828"} Jul 06 20:33:16.837010 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.836893 2564 scope.go:117] "RemoveContainer" containerID="0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545" Jul 06 20:33:16.846576 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.846562 2564 scope.go:117] "RemoveContainer" containerID="0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545" Jul 06 20:33:16.846831 ip-10-0-140-110 kubenswrapper[2564]: E0706 20:33:16.846812 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545\": container with ID starting with 0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545 not found: ID does not exist" containerID="0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545" Jul 06 20:33:16.846908 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.846839 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545"} err="failed to get container status \"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545\": rpc error: code = NotFound desc = could not find container \"0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545\": container with ID starting with 0588d21cd7d495d8bcd7681726818d1b776033ab7c433258391dfd3fb1e50545 not found: ID does not exist" Jul 06 20:33:16.860307 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.860285 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:33:16.865607 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:16.865585 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-a90d79-54d57d9485-wkrfg"] Jul 06 20:33:17.713736 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:17.713707 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7124bba1-8248-43d0-b6d7-51c63afe1a1c" path="/var/lib/kubelet/pods/7124bba1-8248-43d0-b6d7-51c63afe1a1c/volumes" Jul 06 20:33:17.714205 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:17.714144 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f00f03-4d58-43d4-93c1-551d4257d2ea" path="/var/lib/kubelet/pods/84f00f03-4d58-43d4-93c1-551d4257d2ea/volumes" Jul 06 20:33:28.178240 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.178207 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68cb9b94f4-8bs7f_7900e681-d48f-46a3-8e62-0725249f6e7f/maas-api/0.log" Jul 06 20:33:28.290376 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.290355 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722815-8vtvn_a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6/cleanup/0.log" Jul 06 20:33:28.400953 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.400922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722830-mkqdh_edd1039d-140c-4bb3-a9a1-dc28775229b8/cleanup/0.log" Jul 06 20:33:28.517037 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.516965 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9459f97d5-xmd49_5469d566-795f-4a6d-aea8-eb8bfe112cbb/manager/0.log" Jul 06 20:33:28.959423 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.959399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549b57869c-bw2qg_a3ab030c-f8ed-4d14-9ef4-c2189338494f/copy-manifests/0.log" Jul 06 20:33:28.968097 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:28.968081 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549b57869c-bw2qg_a3ab030c-f8ed-4d14-9ef4-c2189338494f/manager/0.log" Jul 06 20:33:29.351180 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:29.351109 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-j7km9_05aa6ab6-290a-4a93-8bcf-604557f681e0/postgres/0.log" Jul 06 20:33:32.008087 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:32.008058 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gfptt_dd3f3568-289c-4fa9-9247-3f5988173f42/discovery/0.log" Jul 06 20:33:33.033204 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:33.033172 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nvbvm_eac879da-4cb3-4323-9d5e-ef9b767752a6/main/0.log" Jul 06 20:33:33.146453 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:33.146426 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fpwh8_fb168aef-c11f-4584-9018-b0b56835d475/main/0.log" Jul 06 20:33:40.721880 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:40.721843 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t5w56_7e613ae5-5cda-463a-8a25-3b357a21de22/global-pull-secret-syncer/0.log" Jul 06 20:33:40.858567 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:40.858530 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vflbj_c4aa3bdd-a39d-453c-ad92-e1f0f31f41e7/konnectivity-agent/0.log" Jul 06 20:33:40.932726 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:40.932702 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-110.ec2.internal_813e38510b3ec379e921946926d00ffe/haproxy/0.log" Jul 06 20:33:48.903257 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:48.903236 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/alertmanager/0.log" Jul 06 20:33:48.933693 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:48.933676 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/config-reloader/0.log" Jul 06 20:33:48.964789 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:48.964756 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/kube-rbac-proxy-web/0.log" Jul 06 20:33:48.988678 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:48.988659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/kube-rbac-proxy/0.log" Jul 06 20:33:49.013554 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.013527 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/kube-rbac-proxy-metric/0.log" Jul 06 20:33:49.042536 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.042512 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/prom-label-proxy/0.log" Jul 06 20:33:49.075044 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.075030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_22533833-fd51-498d-8493-589796869376/init-config-reloader/0.log" Jul 06 20:33:49.240611 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.240519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85f5fbfd6d-48sr6_17b08d44-62c2-43e2-9536-b378b5c4c7e6/metrics-server/0.log" Jul 06 20:33:49.569150 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.569083 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlw2j_1bdc1a9d-0481-4dfa-9989-0aac73e2c444/node-exporter/0.log" Jul 06 20:33:49.645473 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.645446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlw2j_1bdc1a9d-0481-4dfa-9989-0aac73e2c444/kube-rbac-proxy/0.log" Jul 06 20:33:49.696115 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.696095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlw2j_1bdc1a9d-0481-4dfa-9989-0aac73e2c444/init-textfile/0.log" Jul 06 20:33:49.845568 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.845470 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/prometheus/0.log" Jul 06 20:33:49.882084 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.882052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/config-reloader/0.log" Jul 06 20:33:49.908566 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.908540 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/thanos-sidecar/0.log" Jul 06 20:33:49.939728 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.939704 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/kube-rbac-proxy-web/0.log" Jul 06 20:33:49.971462 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.971443 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/kube-rbac-proxy/0.log" Jul 06 20:33:49.999100 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:49.999076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/kube-rbac-proxy-thanos/0.log" Jul 06 20:33:50.030839 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.030816 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d8740e8d-7981-47fc-bcc9-6745f3e84fca/init-config-reloader/0.log" Jul 06 20:33:50.070494 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.070471 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r67v6_760167f0-e709-4890-be0f-9c4d9ca2a2f9/prometheus-operator/0.log" Jul 06 20:33:50.113076 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.113058 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r67v6_760167f0-e709-4890-be0f-9c4d9ca2a2f9/kube-rbac-proxy/0.log" Jul 06 20:33:50.180540 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.180524 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5845c9cd78-t4cn8_a53dd33d-2495-415d-8bba-c0735f0d8907/telemeter-client/0.log" Jul 06 20:33:50.209260 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.209242 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5845c9cd78-t4cn8_a53dd33d-2495-415d-8bba-c0735f0d8907/reload/0.log" Jul 06 20:33:50.238276 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.238258 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5845c9cd78-t4cn8_a53dd33d-2495-415d-8bba-c0735f0d8907/kube-rbac-proxy/0.log" Jul 06 20:33:50.893268 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893235 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q"] Jul 06 20:33:50.893597 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893586 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7124bba1-8248-43d0-b6d7-51c63afe1a1c" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893599 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7124bba1-8248-43d0-b6d7-51c63afe1a1c" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893609 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edd1039d-140c-4bb3-a9a1-dc28775229b8" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893614 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="edd1039d-140c-4bb3-a9a1-dc28775229b8" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893623 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9764620-e05a-43ed-bb91-d55b5c718b1f" containerName="maas-api" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893628 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9764620-e05a-43ed-bb91-d55b5c718b1f" containerName="maas-api" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893635 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893640 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" containerName="cleanup" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893652 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f00f03-4d58-43d4-93c1-551d4257d2ea" containerName="maas-api" Jul 06 20:33:50.893682 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893656 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f00f03-4d58-43d4-93c1-551d4257d2ea" containerName="maas-api" Jul 06 20:33:50.893985 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893713 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f00f03-4d58-43d4-93c1-551d4257d2ea" containerName="maas-api" Jul 06 20:33:50.893985 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893728 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9764620-e05a-43ed-bb91-d55b5c718b1f" containerName="maas-api" Jul 06 20:33:50.893985 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893738 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="edd1039d-140c-4bb3-a9a1-dc28775229b8" containerName="cleanup" Jul 06 20:33:50.893985 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893746 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7124bba1-8248-43d0-b6d7-51c63afe1a1c" containerName="cleanup" Jul 06 20:33:50.893985 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.893752 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b0b74a-17ee-45e2-a3a2-22661fbff7fd" containerName="cleanup" Jul 06 20:33:50.896827 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.896808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:50.899112 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.899093 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jzwh\"/\"kube-root-ca.crt\"" Jul 06 20:33:50.902606 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.902586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jzwh\"/\"openshift-service-ca.crt\"" Jul 06 20:33:50.902763 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.902594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jzwh\"/\"default-dockercfg-k87kc\"" Jul 06 20:33:50.909747 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.909719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q"] Jul 06 20:33:50.944029 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.944002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-podres\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:50.944165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.944051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhhd\" (UniqueName: \"kubernetes.io/projected/6f5a68ab-b890-40bd-abf6-8e78f77f0891-kube-api-access-fjhhd\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:50.944165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.944082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-lib-modules\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:50.944165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.944127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-sys\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:50.944165 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:50.944157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-proc\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045402 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-podres\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045544 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhhd\" (UniqueName: \"kubernetes.io/projected/6f5a68ab-b890-40bd-abf6-8e78f77f0891-kube-api-access-fjhhd\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045544 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-lib-modules\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045544 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-sys\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045681 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045559 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-podres\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045681 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-proc\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045681 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045616 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-sys\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045681 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045622 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-lib-modules\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.045681 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.045656 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6f5a68ab-b890-40bd-abf6-8e78f77f0891-proc\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.054502 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.054481 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhhd\" (UniqueName: \"kubernetes.io/projected/6f5a68ab-b890-40bd-abf6-8e78f77f0891-kube-api-access-fjhhd\") pod \"perf-node-gather-daemonset-tgl5q\" (UID: \"6f5a68ab-b890-40bd-abf6-8e78f77f0891\") " pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.207342 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.207260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:51.337822 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.337796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q"] Jul 06 20:33:51.340520 ip-10-0-140-110 kubenswrapper[2564]: W0706 20:33:51.340496 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f5a68ab_b890_40bd_abf6_8e78f77f0891.slice/crio-1df5788bf1710a676c08dccf2119cacfe976f631ae9cfafab4a613835313742f WatchSource:0}: Error finding container 1df5788bf1710a676c08dccf2119cacfe976f631ae9cfafab4a613835313742f: Status 404 returned error can't find the container with id 1df5788bf1710a676c08dccf2119cacfe976f631ae9cfafab4a613835313742f Jul 06 20:33:51.575271 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:51.575207 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-7vnrm_81cf6450-b4ab-44ae-80b9-66eb53ff6960/networking-console-plugin/0.log" Jul 06 20:33:52.000625 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.000589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" event={"ID":"6f5a68ab-b890-40bd-abf6-8e78f77f0891","Type":"ContainerStarted","Data":"0cd706c5d9fa257c18ed6a9d01deafcbd71479a55f0be06934c838786330ee98"} Jul 06 20:33:52.001076 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.000649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" event={"ID":"6f5a68ab-b890-40bd-abf6-8e78f77f0891","Type":"ContainerStarted","Data":"1df5788bf1710a676c08dccf2119cacfe976f631ae9cfafab4a613835313742f"} Jul 06 20:33:52.001076 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.000752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:52.020296 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.020249 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" podStartSLOduration=2.02023617 podStartE2EDuration="2.02023617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:33:52.018783369 +0000 UTC m=+2554.859785843" watchObservedRunningTime="2026-07-06 20:33:52.02023617 +0000 UTC m=+2554.861238643" Jul 06 20:33:52.665472 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.665444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cd4ff54bc-rxzb6_45d7c16c-c22c-43b3-9daa-5327482719d8/console/0.log" Jul 06 20:33:52.704478 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:52.704452 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-mp9zf_e50a3ad7-d449-4194-aa81-e07043b8b8f7/download-server/0.log" Jul 06 20:33:54.666248 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:54.666218 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9hcp_28df4f4f-a5fa-4718-b59f-59b42a080274/dns/0.log" Jul 06 20:33:54.709487 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:54.709467 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r9hcp_28df4f4f-a5fa-4718-b59f-59b42a080274/kube-rbac-proxy/0.log" Jul 06 20:33:54.806418 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:54.806391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5q9w2_603d4e89-5bc7-4145-8fef-85bf2976fc6b/dns-node-resolver/0.log" Jul 06 20:33:55.439167 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:55.439137 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f5cc55879-wkplg_394a8437-463d-4eab-a16e-4b13247b67f8/registry/0.log" Jul 06 20:33:55.491267 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:55.491238 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s8jgg_0e347eae-c86f-4c0c-a8d5-d191b34e1048/node-ca/0.log" Jul 06 20:33:56.751957 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:56.751924 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gfptt_dd3f3568-289c-4fa9-9247-3f5988173f42/discovery/0.log" Jul 06 20:33:58.007023 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:58.006950 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rhkcb_2cb88d98-3693-4563-a9e2-99777a639030/serve-healthcheck-canary/0.log" Jul 06 20:33:58.014766 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:58.014738 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5jzwh/perf-node-gather-daemonset-tgl5q" Jul 06 20:33:58.901163 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:58.901134 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxdd8_cda2f75a-466d-42c6-b86a-de44f19e42b4/kube-rbac-proxy/0.log" Jul 06 20:33:58.937166 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:58.937134 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxdd8_cda2f75a-466d-42c6-b86a-de44f19e42b4/exporter/0.log" Jul 06 20:33:58.972357 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:33:58.972331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxdd8_cda2f75a-466d-42c6-b86a-de44f19e42b4/extractor/0.log" Jul 06 20:34:01.964021 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:01.963984 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68cb9b94f4-8bs7f_7900e681-d48f-46a3-8e62-0725249f6e7f/maas-api/0.log" Jul 06 20:34:01.990828 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:01.990796 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722815-8vtvn_a1c7d4ef-0b7c-4954-8ffc-812bf9a324d6/cleanup/0.log" Jul 06 20:34:02.036942 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.036914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722830-mkqdh_edd1039d-140c-4bb3-a9a1-dc28775229b8/cleanup/0.log" Jul 06 20:34:02.092349 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.092303 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gfptt_dd3f3568-289c-4fa9-9247-3f5988173f42/discovery/0.log" Jul 06 20:34:02.160119 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.160084 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9459f97d5-xmd49_5469d566-795f-4a6d-aea8-eb8bfe112cbb/manager/0.log" Jul 06 20:34:02.387969 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.387936 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549b57869c-bw2qg_a3ab030c-f8ed-4d14-9ef4-c2189338494f/manager/0.log" Jul 06 20:34:02.422614 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.422590 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549b57869c-bw2qg_a3ab030c-f8ed-4d14-9ef4-c2189338494f/copy-manifests/0.log" Jul 06 20:34:02.682299 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:02.682270 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-j7km9_05aa6ab6-290a-4a93-8bcf-604557f681e0/postgres/0.log" Jul 06 20:34:03.774448 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:03.774411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gfptt_dd3f3568-289c-4fa9-9247-3f5988173f42/discovery/0.log" Jul 06 20:34:04.145478 ip-10-0-140-110 kubenswrapper[2564]: I0706 20:34:04.145444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c876dc4c4-jnmf4_ae323d21-5d59-4feb-b4b4-7c545d05a673/manager/0.log"