Jul 09 18:53:43.060210 ip-10-0-131-3 systemd[1]: Starting Kubernetes Kubelet... Jul 09 18:53:43.439310 ip-10-0-131-3 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:43.439829 ip-10-0-131-3 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 18:53:43.439829 ip-10-0-131-3 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:43.439829 ip-10-0-131-3 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 18:53:43.439829 ip-10-0-131-3 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:53:43.439829 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.439390 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 18:53:43.447105 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447077 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.447207 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447199 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.447207 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447206 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447210 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447213 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447216 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447218 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447221 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447227 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447231 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447234 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447237 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447240 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447243 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447246 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447249 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447252 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447254 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447257 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447261 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447264 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447267 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.447271 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447269 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447272 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447275 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447278 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447281 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447284 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447286 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447289 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447291 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447294 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447296 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447299 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447302 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447305 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447307 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447310 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447312 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447315 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447317 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.447746 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447320 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447322 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447325 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447327 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447330 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447332 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447335 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447337 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447339 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447342 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447347 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447367 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447370 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447373 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447378 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447381 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447383 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447386 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447388 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447391 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.448229 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447394 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447396 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447399 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447401 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447404 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447406 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447409 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447411 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447415 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447419 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447422 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447425 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447428 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447430 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447433 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447436 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447439 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447441 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447444 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447446 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.448703 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447449 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447452 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447463 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447467 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447470 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447474 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447477 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447480 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447483 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447486 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447489 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447491 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447494 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447496 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447499 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447502 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447505 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.447507 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448273 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.449196 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448281 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448284 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448287 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448289 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448292 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448295 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448298 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448300 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448303 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448306 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448308 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448311 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448314 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448316 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448319 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448321 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448326 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448328 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448331 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448333 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.449653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448336 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448339 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448341 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448344 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448347 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448349 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448352 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448354 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448357 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448360 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448362 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448365 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448367 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448370 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448373 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448375 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448378 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448381 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448383 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448386 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.450141 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448388 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448391 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448393 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448396 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448398 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448401 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448403 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448406 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448409 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448413 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448415 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448417 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448420 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448422 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448425 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448427 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448430 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448432 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448435 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448437 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.450595 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448440 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448442 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448445 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448448 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448450 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448453 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448456 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448459 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448462 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448465 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448468 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448470 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448473 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448475 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448478 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448480 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448483 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448485 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448488 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.451062 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448492 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448496 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448500 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448504 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448508 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448510 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448513 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448517 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448521 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448524 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448526 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448530 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448533 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448536 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448538 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448541 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448544 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448546 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448549 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.451505 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448953 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448958 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448961 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448964 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448967 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448970 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448973 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448975 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448978 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448981 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448984 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448986 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448989 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448991 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448994 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.448996 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449000 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449002 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449005 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449008 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.451978 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449010 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449013 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449015 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449018 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449022 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449025 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449028 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449030 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449033 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449036 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449038 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449041 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449043 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449046 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449048 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449051 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449054 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449056 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449060 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449062 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.452465 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449065 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449068 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449070 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449072 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449075 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449078 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449082 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449084 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449087 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449089 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449092 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449095 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449097 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449100 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449102 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449105 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449108 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449110 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449113 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449115 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.452945 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449117 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449120 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449123 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449126 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449128 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449131 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449134 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449136 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449139 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449141 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449144 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449146 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449149 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449151 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449154 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449156 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449159 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449161 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449164 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449166 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.453456 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449169 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449171 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449174 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449177 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449179 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449182 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449184 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449190 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449193 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449196 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449199 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449202 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449205 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449212 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449216 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449222 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449225 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449228 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449231 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.453934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449298 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449305 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449313 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449317 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449321 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449324 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449329 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449333 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449336 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449339 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449342 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449346 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449349 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449351 2578 flags.go:64] FLAG: --cgroup-root="" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449354 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449357 2578 flags.go:64] FLAG: --client-ca-file="" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449361 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449364 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449368 2578 flags.go:64] FLAG: --cluster-domain="" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449371 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449374 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449377 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449380 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449384 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 18:53:43.454370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449387 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449390 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449395 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449397 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449402 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449404 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449408 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449410 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449414 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449418 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449433 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449436 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449439 2578 flags.go:64] FLAG: --enable-server="true" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449442 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449446 2578 flags.go:64] FLAG: --event-burst="100" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449449 2578 flags.go:64] FLAG: --event-qps="50" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449452 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449455 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449457 2578 flags.go:64] FLAG: --eviction-hard="" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449461 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449464 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449466 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449469 2578 flags.go:64] FLAG: --eviction-soft="" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449472 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449474 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 18:53:43.455031 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449478 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449481 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449483 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449486 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449489 2578 flags.go:64] FLAG: --feature-gates="" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449492 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449495 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449498 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449501 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449505 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449508 2578 flags.go:64] FLAG: --help="false" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449512 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-3.ec2.internal" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449515 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449519 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449521 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449524 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449528 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449531 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449533 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449536 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449539 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449542 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449545 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449547 2578 flags.go:64] FLAG: --kube-reserved="" Jul 09 18:53:43.455609 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449550 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449553 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449556 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449558 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449561 2578 flags.go:64] FLAG: --lock-file="" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449564 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449566 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449569 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449574 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449577 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449580 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449583 2578 flags.go:64] FLAG: --logging-format="text" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449585 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449588 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449591 2578 flags.go:64] FLAG: --manifest-url="" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449594 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449598 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449602 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449606 2578 flags.go:64] FLAG: --max-pods="110" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449610 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449613 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449615 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449618 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449621 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449624 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 18:53:43.456193 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449626 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449633 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449636 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449639 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449642 2578 flags.go:64] FLAG: --pod-cidr="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449645 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449650 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449653 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449656 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449660 2578 flags.go:64] FLAG: --port="10250" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449663 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449665 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07546ed8d9e500f46" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449669 2578 flags.go:64] FLAG: --qos-reserved="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449671 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449674 2578 flags.go:64] FLAG: --register-node="true" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449677 2578 flags.go:64] FLAG: --register-with-taints="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449680 2578 flags.go:64] FLAG: --registry-burst="10" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449684 2578 flags.go:64] FLAG: --registry-qps="5" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449686 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449689 2578 flags.go:64] FLAG: --reserved-memory="" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449693 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449696 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449698 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449701 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449704 2578 flags.go:64] FLAG: --runonce="false" Jul 09 18:53:43.456800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449708 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449711 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449714 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449717 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449720 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449723 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449725 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449728 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449731 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449734 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449737 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449740 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449742 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449746 2578 flags.go:64] FLAG: --system-cgroups="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449748 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449753 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449756 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449759 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449762 2578 flags.go:64] FLAG: --tls-min-version="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449765 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449767 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449770 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449773 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449776 2578 flags.go:64] FLAG: --v="2" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449779 2578 flags.go:64] FLAG: --version="false" Jul 09 18:53:43.457432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449784 2578 flags.go:64] FLAG: --vmodule="" Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449788 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.449791 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449900 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449904 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449907 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449910 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449914 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449931 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449934 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449937 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449940 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449942 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449945 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449948 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449951 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449953 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449956 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449959 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449961 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.458045 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449964 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449966 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449969 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449972 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449974 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449976 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449979 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449981 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449984 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449986 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449988 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449991 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449993 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449996 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.449999 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450001 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450004 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450006 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450009 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450013 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.458557 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450016 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450018 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450021 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450023 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450026 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450028 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450031 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450035 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450039 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450041 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450044 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450047 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450049 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450051 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450055 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450059 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450062 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450065 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450068 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.459090 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450070 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450073 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450075 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450078 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450080 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450083 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450086 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450088 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450091 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450094 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450096 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450098 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450103 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450105 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450108 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450111 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450114 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450116 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450119 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.459516 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450121 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450124 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450128 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450131 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450134 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450137 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450139 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450142 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450145 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450148 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450151 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450154 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450157 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450159 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450162 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450165 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450167 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450170 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450173 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450175 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.459977 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450177 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450180 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450184 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.450187 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.450830 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.457655 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.457670 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.458507 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.458514 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458575 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458580 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458583 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458586 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458589 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458592 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.460470 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458594 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458597 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458599 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458602 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458604 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458607 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458609 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458612 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458615 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458618 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458620 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458623 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458625 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458627 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458630 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458632 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458635 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458637 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458639 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458642 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.460842 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458645 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458647 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458649 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458654 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458656 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458659 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458661 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458664 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458666 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458669 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458671 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458674 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458676 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458680 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458684 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458686 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458689 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458693 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458696 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.461345 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458699 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458702 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458705 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458708 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458710 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458713 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458715 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458718 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458720 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458723 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458725 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458728 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458731 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458733 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458736 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458738 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458741 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458744 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458747 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458750 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.461809 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458752 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458755 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458757 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458760 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458762 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458765 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458767 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458770 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458772 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458774 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458777 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458779 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458781 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458784 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458788 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458791 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458794 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458797 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458799 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458802 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.462290 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458804 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458806 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458809 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458811 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458814 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458816 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458819 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458821 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458825 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458827 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458830 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458833 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458835 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458837 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.458842 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:43.462793 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458968 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458973 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458976 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458979 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458982 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458984 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458987 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458990 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458992 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458995 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.458998 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459000 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459003 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459005 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459008 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459010 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459013 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459015 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459017 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459020 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:53:43.463208 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459022 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459025 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459027 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459029 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459032 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459034 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459037 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459040 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459045 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459048 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459050 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459053 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459055 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459057 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459060 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459062 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459065 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459068 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459072 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459074 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:53:43.463677 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459077 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459079 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459082 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459084 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459086 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459089 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459091 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459094 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459097 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459099 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459102 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459105 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459107 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459110 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459112 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459114 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459117 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459119 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459122 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459124 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:53:43.464155 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459127 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459130 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459132 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459135 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459137 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459140 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459142 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459145 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459147 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459149 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459152 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459154 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459156 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459159 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459161 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459165 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459168 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459171 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459173 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:53:43.464653 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459176 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459178 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459182 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459185 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459204 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459207 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459210 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459213 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459216 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459219 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459222 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459225 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459227 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459230 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459233 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459235 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459238 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459241 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459243 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:53:43.465115 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:43.459246 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.459251 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.459376 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.461325 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.462724 2578 server.go:1019] "Starting client certificate rotation" Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.462822 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.462851 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:53:43.465600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.465225 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:53:43.470704 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.470688 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:53:43.482953 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.482933 2578 log.go:25] "Validated CRI v1 runtime API" Jul 09 18:53:43.484861 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.484847 2578 log.go:25] "Validated CRI v1 image API" Jul 09 18:53:43.486338 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.486323 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 18:53:43.488803 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.488783 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87605875-df39-4242-92af-c2399791fef3:/dev/nvme0n1p4 89fd6a46-6716-472e-b579-2afdd8924a53:/dev/nvme0n1p3] Jul 09 18:53:43.488869 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:43.488802 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 18:53:44.905762 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.905743 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:53:44.906684 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.906577 2578 manager.go:216] Machine: {Timestamp:2026-07-09 18:53:44.905139703 +0000 UTC m=+0.321854104 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3180522 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bc746d4a9656871665ff40bc5d376 SystemUUID:ec2bc746-d4a9-6568-7166-5ff40bc5d376 BootID:80de8e46-2ac6-473c-861d-5beec47c0ae2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:e3:eb:c5:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:e3:eb:c5:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:89:fa:ab:7e:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 18:53:44.906684 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.906680 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 18:53:44.906811 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.906769 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 18:53:44.909568 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.909546 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 18:53:44.909708 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.909569 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 18:53:44.910936 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.910927 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 18:53:44.910979 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.910939 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 18:53:44.910979 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.910947 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:53:44.911652 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.911643 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:53:44.912927 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.912917 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 18:53:44.915689 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915678 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:53:44.915848 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915838 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 18:53:44.915877 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915868 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 09 18:53:44.915912 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915877 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 18:53:44.915912 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915894 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 18:53:44.915912 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915903 2578 kubelet.go:398] "Adding apiserver pod source" Jul 09 18:53:44.916011 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.915918 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 18:53:44.916268 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.916251 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:53:44.916338 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.916273 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:53:44.919069 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.919054 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 18:53:44.920896 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.920882 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 18:53:44.920951 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.920912 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 18:53:44.921080 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921069 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 18:53:44.921111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921086 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 18:53:44.921111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921091 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 18:53:44.921111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921097 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 18:53:44.921111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921105 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 18:53:44.921111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921110 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 18:53:44.921628 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921619 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 18:53:44.921628 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921628 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 18:53:44.921683 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921634 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 18:53:44.921683 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921640 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 18:53:44.921683 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921647 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 18:53:44.921683 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.921655 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 18:53:44.922504 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.922495 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 18:53:44.922504 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.922503 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 18:53:44.925576 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.925554 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 18:53:44.925671 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.925627 2578 server.go:1268] "Started kubelet" Jul 09 18:53:44.925671 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.925660 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 18:53:44.925805 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.925663 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 18:53:44.925805 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.925705 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 18:53:44.925805 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.925766 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 18:53:44.925907 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.925810 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 18:53:44.926906 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.926889 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 18:53:44.927177 ip-10-0-131-3 systemd[1]: Started Kubernetes Kubelet. Jul 09 18:53:44.932589 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.932560 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 09 18:53:44.932823 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.932795 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 18:53:44.933256 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.933234 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm9xg" Jul 09 18:53:44.934492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.934472 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 18:53:44.934492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.934483 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 18:53:44.934691 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.934672 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:44.935152 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935136 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 18:53:44.935240 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935139 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 18:53:44.935240 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935167 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 18:53:44.935333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935267 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 09 18:53:44.935333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935276 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 09 18:53:44.935333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935300 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 18:53:44.935333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935313 2578 factory.go:55] Registering systemd factory Jul 09 18:53:44.935333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935323 2578 factory.go:223] Registration of the systemd container factory successfully Jul 09 18:53:44.935573 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935528 2578 factory.go:153] Registering CRI-O factory Jul 09 18:53:44.935573 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935540 2578 factory.go:223] Registration of the crio container factory successfully Jul 09 18:53:44.935573 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935560 2578 factory.go:103] Registering Raw factory Jul 09 18:53:44.935573 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.935573 2578 manager.go:1196] Started watching for new ooms in manager Jul 09 18:53:44.936217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.936201 2578 manager.go:349] Starting recovery of all containers Jul 09 18:53:44.936405 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.935449 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-3.ec2.internal.18c0b3df7ed3cae5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-3.ec2.internal,UID:ip-10-0-131-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-3.ec2.internal,},FirstTimestamp:2026-07-09 18:53:44.925604581 +0000 UTC m=+0.342318988,LastTimestamp:2026-07-09 18:53:44.925604581 +0000 UTC m=+0.342318988,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-3.ec2.internal,}" Jul 09 18:53:44.941891 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.941868 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bm9xg" Jul 09 18:53:44.942148 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.942121 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 18:53:44.942330 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.942290 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 18:53:44.944825 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.944804 2578 manager.go:354] Recovery completed Jul 09 18:53:44.949665 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.949654 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:44.952632 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.952531 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:44.952678 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.952644 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:44.952678 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.952655 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:44.953095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.953079 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 18:53:44.953095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.953091 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:44.953212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.953092 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 18:53:44.953212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.953160 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:53:44.954188 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954172 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:44.954244 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954199 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:44.954244 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954211 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:44.954349 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954339 2578 policy_none.go:49] "None policy: Start" Jul 09 18:53:44.954384 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954355 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 18:53:44.954384 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.954368 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 18:53:44.955170 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.955117 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-3.ec2.internal.18c0b3df8070314d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-3.ec2.internal,UID:ip-10-0-131-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-3.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-3.ec2.internal,},FirstTimestamp:2026-07-09 18:53:44.952631629 +0000 UTC m=+0.369346029,LastTimestamp:2026-07-09 18:53:44.952631629 +0000 UTC m=+0.369346029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-3.ec2.internal,}" Jul 09 18:53:44.955624 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.955612 2578 policy_none.go:47] "Start" Jul 09 18:53:44.988288 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988272 2578 manager.go:337] "Starting Device Plugin manager" Jul 09 18:53:44.988360 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.988307 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 18:53:44.988360 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988316 2578 server.go:87] "Starting device plugin registration server" Jul 09 18:53:44.988538 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988525 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 18:53:44.988580 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988539 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 18:53:44.988642 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988620 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 18:53:44.988744 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988711 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 18:53:44.988744 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:44.988742 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 18:53:44.989304 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.989273 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 18:53:44.989362 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:44.989340 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.089186 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.089162 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:45.090296 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.090274 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:45.090359 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.090307 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:45.090359 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.090319 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:45.090359 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.090343 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.098489 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.098456 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 18:53:45.099658 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.099636 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 18:53:45.099743 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.099685 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 18:53:45.099743 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.099710 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 18:53:45.099822 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.099766 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 18:53:45.100354 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.100336 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.100442 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.100363 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-3.ec2.internal\": node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.102622 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.102580 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:45.124002 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.123981 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.200030 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.200005 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal"] Jul 09 18:53:45.200158 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.200082 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:45.201412 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.201397 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:45.201492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.201427 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:45.201492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.201440 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:45.203336 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.203323 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:45.203458 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.203444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.203508 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.203479 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:45.204179 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204165 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:45.204249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204191 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:45.204249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204200 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:45.204830 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204815 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:45.204880 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204843 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:45.204880 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.204856 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:45.205391 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.205378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.205431 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.205401 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:53:45.206012 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.205997 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:53:45.206084 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.206021 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:53:45.206084 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.206033 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:53:45.222229 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.222210 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-3.ec2.internal\" not found" node="ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.224037 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.224020 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.225498 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.225485 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-3.ec2.internal\" not found" node="ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.236685 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.236669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.236775 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.236697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.236775 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.236721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7acd9c4c33c3ec7d300bea1511a846a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-3.ec2.internal\" (UID: \"7acd9c4c33c3ec7d300bea1511a846a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.324851 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.324830 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.337412 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.337485 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.337485 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7acd9c4c33c3ec7d300bea1511a846a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-3.ec2.internal\" (UID: \"7acd9c4c33c3ec7d300bea1511a846a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.337485 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7acd9c4c33c3ec7d300bea1511a846a1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-3.ec2.internal\" (UID: \"7acd9c4c33c3ec7d300bea1511a846a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.337603 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337501 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.337603 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.337502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8267dc650a784c1be2605ae2dc268fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal\" (UID: \"a8267dc650a784c1be2605ae2dc268fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.425661 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.425599 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.524240 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.524199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.526315 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.526297 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.528279 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.528258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:45.626418 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.626383 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.726908 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.726836 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.827562 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.827523 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.865563 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.865537 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:45.874254 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.874239 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 18:53:45.874355 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.874339 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:53:45.874401 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.874384 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:53:45.927742 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:45.927711 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:45.945710 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.945676 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 18:48:44 +0000 UTC" deadline="2028-04-04 16:28:32.472056762 +0000 UTC" Jul 09 18:53:45.945710 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:45.945707 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15237h34m46.526352256s" Jul 09 18:53:46.028062 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.027996 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:46.099138 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:46.099101 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8267dc650a784c1be2605ae2dc268fa.slice/crio-a7c89323fda6f43dca973f5a3a89e17245dbee6733bb2e44886dee3f44389aec WatchSource:0}: Error finding container a7c89323fda6f43dca973f5a3a89e17245dbee6733bb2e44886dee3f44389aec: Status 404 returned error can't find the container with id a7c89323fda6f43dca973f5a3a89e17245dbee6733bb2e44886dee3f44389aec Jul 09 18:53:46.099662 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:46.099644 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acd9c4c33c3ec7d300bea1511a846a1.slice/crio-56eb49a10ca93d65f751b4335997ba62536c2de9368a96866d41ecc5b2e4e24f WatchSource:0}: Error finding container 56eb49a10ca93d65f751b4335997ba62536c2de9368a96866d41ecc5b2e4e24f: Status 404 returned error can't find the container with id 56eb49a10ca93d65f751b4335997ba62536c2de9368a96866d41ecc5b2e4e24f Jul 09 18:53:46.103939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.103925 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:53:46.128309 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.128289 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:46.169445 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.169423 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:46.229251 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.229214 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:46.329756 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.329680 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-3.ec2.internal\" not found" Jul 09 18:53:46.349203 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.349182 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:46.434794 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.434766 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" Jul 09 18:53:46.445310 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.445292 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:53:46.446341 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.446328 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" Jul 09 18:53:46.459800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.459784 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:53:46.917407 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.917363 2578 apiserver.go:52] "Watching apiserver" Jul 09 18:53:46.930814 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.930784 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 18:53:46.932911 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.932874 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dwv46","openshift-image-registry/node-ca-n9jq9","openshift-multus/multus-additional-cni-plugins-sdfpx","openshift-network-operator/iptables-alerter-w8pjl","openshift-ovn-kubernetes/ovnkube-node-x4hc8","kube-system/global-pull-secret-syncer-g2nbg","kube-system/konnectivity-agent-pc4s2","kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal","openshift-dns/node-resolver-79j55","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal","openshift-multus/multus-l5tbn","openshift-multus/network-metrics-daemon-f549x","openshift-network-diagnostics/network-check-target-7hvxv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb"] Jul 09 18:53:46.935568 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.935504 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 18:53:46.935568 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.935516 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:46.937814 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.937758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:46.937921 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.937841 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:46.940590 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.940343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:46.941566 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.941547 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 18:53:46.941654 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.941633 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 18:53:46.941798 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.941781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwhcx\"" Jul 09 18:53:46.942167 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.942150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:46.944599 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.944578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:46.947352 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.947330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-agent-certs\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:46.947433 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.947365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-konnectivity-ca\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:46.947433 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.947391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:46.947526 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.947498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:46.947587 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.947568 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:46.948792 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.948769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 18:53:46.949475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.949455 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 18:53:46.949798 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.949775 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.949994 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.949974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:46.952148 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.952124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:46.954375 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.954349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:46.957061 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:46.957559 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957536 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6q6f\"" Jul 09 18:53:46.958171 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957700 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rwldj\"" Jul 09 18:53:46.958171 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957763 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 18:53:46.958171 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957937 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.958171 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.957954 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 18:53:46.958171 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.958116 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.958483 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.958214 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.959750 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.959399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5tbn" Jul 09 18:53:46.960810 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.960790 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 18:53:46.961890 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.961872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:46.961978 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:46.961935 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:46.962445 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.962425 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 18:53:46.962581 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.962563 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lmw5k\"" Jul 09 18:53:46.962670 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.962653 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 18:53:46.962748 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.962710 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.962825 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.962808 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.963178 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.963155 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.963178 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.963174 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.963302 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.963200 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.963352 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.963340 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.963440 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.963423 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.964227 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964045 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 18:53:46.964227 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964056 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wdmng\"" Jul 09 18:53:46.964227 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964196 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.964401 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964289 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 18:53:46.964459 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964413 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 18:53:46.964814 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964630 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fctnq\"" Jul 09 18:53:46.964912 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.964886 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 18:53:46.967125 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.966003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jt29m\"" Jul 09 18:53:46.967125 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.966216 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 18:53:46.967125 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.966601 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcvxf\"" Jul 09 18:53:46.967301 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.967211 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qmfjd\"" Jul 09 18:53:46.969328 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.969305 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 18:53:46.975640 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.975431 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:53:46.979391 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:46.979371 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:47.018409 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.018378 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ls8tg" Jul 09 18:53:47.029687 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.029662 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ls8tg" Jul 09 18:53:47.036494 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.036465 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 18:53:47.047701 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-dbus\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf59030-65db-468f-ba49-ee906a44d696-tmp-dir\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-run\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-daemon-config\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.047892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-systemd\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-socket-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-os-release\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.047961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-script-lib\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysconfig\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-var-lib-kubelet\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-hostroot\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-ovn\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-env-overrides\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnms\" (UniqueName: \"kubernetes.io/projected/6be92434-685d-46cd-bc29-b654ee8218e7-kube-api-access-gnnms\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9559bce9-1971-40aa-8e5d-de6f89e2d607-host\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.048157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86d74512-0c1b-4f1b-b025-8daa776a5c8a-host-slash\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-os-release\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-log-socket\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-config\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf59030-65db-468f-ba49-ee906a44d696-hosts-file\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-host\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-k8s-cni-cncf-io\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-systemd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-conf\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-system-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-netns\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-bin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-multus\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.048562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26xw\" (UniqueName: \"kubernetes.io/projected/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-kube-api-access-g26xw\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-konnectivity-ca\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbss\" (UniqueName: \"kubernetes.io/projected/9559bce9-1971-40aa-8e5d-de6f89e2d607-kube-api-access-hjbss\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-device-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-kubelet\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-etc-kubernetes\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlt7\" (UniqueName: \"kubernetes.io/projected/daf59030-65db-468f-ba49-ee906a44d696-kube-api-access-9hlt7\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-lib-modules\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-sys-fs\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-netns\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nzz\" (UniqueName: \"kubernetes.io/projected/86d74512-0c1b-4f1b-b025-8daa776a5c8a-kube-api-access-q6nzz\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm48\" (UniqueName: \"kubernetes.io/projected/aa94099b-e104-4adb-a96a-e9737f9c043b-kube-api-access-7sm48\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.048987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cnibin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.049216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-kubelet\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-node-log\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-modprobe-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86d74512-0c1b-4f1b-b025-8daa776a5c8a-iptables-alerter-script\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-konnectivity-ca\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-conf-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-agent-certs\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-systemd-units\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-etc-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9559bce9-1971-40aa-8e5d-de6f89e2d607-serviceca\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-cnibin\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.049846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5fs\" (UniqueName: \"kubernetes.io/projected/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-kube-api-access-zg5fs\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-registration-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cni-binary-copy\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049581 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-bin\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-netd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-kubelet-config\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-kubernetes\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-sys\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-etc-tuned\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-tmp\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-multus-certs\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-system-cni-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj89m\" (UniqueName: \"kubernetes.io/projected/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-kube-api-access-gj89m\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-slash\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050461 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-var-lib-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.050990 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.050990 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.050990 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.049996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-socket-dir-parent\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.050990 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.050030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nfs\" (UniqueName: \"kubernetes.io/projected/241032c6-fb35-4a4c-917d-539cc479b8cc-kube-api-access-k8nfs\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.053080 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.053060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7-agent-certs\") pod \"konnectivity-agent-pc4s2\" (UID: \"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7\") " pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:47.058634 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.058617 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:47.058634 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.058637 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:47.058776 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.058647 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:47.058776 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.058697 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.558680328 +0000 UTC m=+2.975394717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:47.104335 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.104289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" event={"ID":"a8267dc650a784c1be2605ae2dc268fa","Type":"ContainerStarted","Data":"a7c89323fda6f43dca973f5a3a89e17245dbee6733bb2e44886dee3f44389aec"} Jul 09 18:53:47.105486 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.105463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" event={"ID":"7acd9c4c33c3ec7d300bea1511a846a1","Type":"ContainerStarted","Data":"56eb49a10ca93d65f751b4335997ba62536c2de9368a96866d41ecc5b2e4e24f"} Jul 09 18:53:47.150762 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysconfig\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-var-lib-kubelet\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-hostroot\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-ovn\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-ovn\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysconfig\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-hostroot\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.150899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-var-lib-kubelet\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-env-overrides\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnms\" (UniqueName: \"kubernetes.io/projected/6be92434-685d-46cd-bc29-b654ee8218e7-kube-api-access-gnnms\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9559bce9-1971-40aa-8e5d-de6f89e2d607-host\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.150998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86d74512-0c1b-4f1b-b025-8daa776a5c8a-host-slash\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-os-release\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-log-socket\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-config\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf59030-65db-468f-ba49-ee906a44d696-hosts-file\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-host\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-k8s-cni-cncf-io\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-systemd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-conf\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-log-socket\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-system-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-netns\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151250 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9559bce9-1971-40aa-8e5d-de6f89e2d607-host\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-bin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86d74512-0c1b-4f1b-b025-8daa776a5c8a-host-slash\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-multus\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26xw\" (UniqueName: \"kubernetes.io/projected/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-kube-api-access-g26xw\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-os-release\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbss\" (UniqueName: \"kubernetes.io/projected/9559bce9-1971-40aa-8e5d-de6f89e2d607-kube-api-access-hjbss\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-device-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151394 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-systemd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-kubelet\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-multus\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-etc-kubernetes\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/daf59030-65db-468f-ba49-ee906a44d696-hosts-file\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-cni-bin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-system-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151335 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-env-overrides\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.151932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-host\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151576 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-etc-kubernetes\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-k8s-cni-cncf-io\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-conf\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-var-lib-kubelet\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-device-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-netns\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlt7\" (UniqueName: \"kubernetes.io/projected/daf59030-65db-468f-ba49-ee906a44d696-kube-api-access-9hlt7\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-config\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-lib-modules\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-sys-fs\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-netns\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nzz\" (UniqueName: \"kubernetes.io/projected/86d74512-0c1b-4f1b-b025-8daa776a5c8a-kube-api-access-q6nzz\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm48\" (UniqueName: \"kubernetes.io/projected/aa94099b-e104-4adb-a96a-e9737f9c043b-kube-api-access-7sm48\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.152778 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cnibin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-kubelet\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-kubelet\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cnibin\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-sysctl-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.152364 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-sys-fs\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.152435 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.652415351 +0000 UTC m=+3.069129758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152611 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-netns\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.151938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-lib-modules\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152718 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152747 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-node-log\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-modprobe-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.153505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86d74512-0c1b-4f1b-b025-8daa776a5c8a-iptables-alerter-script\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-conf-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-systemd-units\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-etc-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9559bce9-1971-40aa-8e5d-de6f89e2d607-serviceca\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-cnibin\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-modprobe-d\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5fs\" (UniqueName: \"kubernetes.io/projected/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-kube-api-access-zg5fs\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-registration-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cni-binary-copy\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-run-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-systemd-units\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-etc-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.152878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-node-log\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-conf-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-bin\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153803 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-bin\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-netd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-kubelet-config\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-kubernetes\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-sys\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-etc-tuned\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.153998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-tmp\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9559bce9-1971-40aa-8e5d-de6f89e2d607-serviceca\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-multus-certs\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154084 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-kubernetes\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154090 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-host-run-multus-certs\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86d74512-0c1b-4f1b-b025-8daa776a5c8a-iptables-alerter-script\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-sys\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154147 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-cni-netd\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.158945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154195 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-kubelet-config\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154233 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-registration-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.154952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/241032c6-fb35-4a4c-917d-539cc479b8cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovn-node-metrics-cert\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-system-cni-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj89m\" (UniqueName: \"kubernetes.io/projected/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-kube-api-access-gj89m\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-system-cni-dir\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-slash\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155214 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/241032c6-fb35-4a4c-917d-539cc479b8cc-cnibin\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-var-lib-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-socket-dir-parent\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nfs\" (UniqueName: \"kubernetes.io/projected/241032c6-fb35-4a4c-917d-539cc479b8cc-kube-api-access-k8nfs\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155343 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-cni-binary-copy\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-dbus\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.159849 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf59030-65db-468f-ba49-ee906a44d696-tmp-dir\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-run\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155436 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-slash\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-daemon-config\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-var-lib-openvswitch\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-socket-dir-parent\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daf59030-65db-468f-ba49-ee906a44d696-tmp-dir\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-systemd\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-socket-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-cni-dir\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.155979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-os-release\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-script-lib\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-socket-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-etc-systemd\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c4b166f-98c3-43d4-a8b6-6872182419f6-dbus\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.160660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156175 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6be92434-685d-46cd-bc29-b654ee8218e7-run\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-os-release\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.156266 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.156330 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.656314644 +0000 UTC m=+3.073029033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-multus-daemon-config\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa94099b-e104-4adb-a96a-e9737f9c043b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-tmp\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.156766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-ovnkube-script-lib\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.161547 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.157060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6be92434-685d-46cd-bc29-b654ee8218e7-etc-tuned\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.162066 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.162045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbss\" (UniqueName: \"kubernetes.io/projected/9559bce9-1971-40aa-8e5d-de6f89e2d607-kube-api-access-hjbss\") pod \"node-ca-n9jq9\" (UID: \"9559bce9-1971-40aa-8e5d-de6f89e2d607\") " pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.166380 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.166273 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm48\" (UniqueName: \"kubernetes.io/projected/aa94099b-e104-4adb-a96a-e9737f9c043b-kube-api-access-7sm48\") pod \"aws-ebs-csi-driver-node-v2pcb\" (UID: \"aa94099b-e104-4adb-a96a-e9737f9c043b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.166918 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.166758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5fs\" (UniqueName: \"kubernetes.io/projected/0824ce37-5f83-4f67-bf0f-56c7f723f3d2-kube-api-access-zg5fs\") pod \"ovnkube-node-x4hc8\" (UID: \"0824ce37-5f83-4f67-bf0f-56c7f723f3d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.166918 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.166864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnms\" (UniqueName: \"kubernetes.io/projected/6be92434-685d-46cd-bc29-b654ee8218e7-kube-api-access-gnnms\") pod \"tuned-dwv46\" (UID: \"6be92434-685d-46cd-bc29-b654ee8218e7\") " pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.167140 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.167108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nzz\" (UniqueName: \"kubernetes.io/projected/86d74512-0c1b-4f1b-b025-8daa776a5c8a-kube-api-access-q6nzz\") pod \"iptables-alerter-w8pjl\" (UID: \"86d74512-0c1b-4f1b-b025-8daa776a5c8a\") " pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.167816 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.167771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26xw\" (UniqueName: \"kubernetes.io/projected/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-kube-api-access-g26xw\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.168223 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.168204 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlt7\" (UniqueName: \"kubernetes.io/projected/daf59030-65db-468f-ba49-ee906a44d696-kube-api-access-9hlt7\") pod \"node-resolver-79j55\" (UID: \"daf59030-65db-468f-ba49-ee906a44d696\") " pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.170837 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.170816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nfs\" (UniqueName: \"kubernetes.io/projected/241032c6-fb35-4a4c-917d-539cc479b8cc-kube-api-access-k8nfs\") pod \"multus-additional-cni-plugins-sdfpx\" (UID: \"241032c6-fb35-4a4c-917d-539cc479b8cc\") " pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.171096 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.171076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj89m\" (UniqueName: \"kubernetes.io/projected/e19f6665-52bb-43cd-8fc8-f3ca82df5cad-kube-api-access-gj89m\") pod \"multus-l5tbn\" (UID: \"e19f6665-52bb-43cd-8fc8-f3ca82df5cad\") " pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.247799 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.247767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:53:47.256473 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.256445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" Jul 09 18:53:47.272317 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.272296 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w8pjl" Jul 09 18:53:47.279048 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.279029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:53:47.286785 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.286763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-79j55" Jul 09 18:53:47.294418 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.294387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dwv46" Jul 09 18:53:47.301293 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.301273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n9jq9" Jul 09 18:53:47.308285 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.308258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" Jul 09 18:53:47.312938 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.312912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5tbn" Jul 09 18:53:47.559075 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.558988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:47.559235 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.559136 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:47.559235 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.559153 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:47.559235 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.559165 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:47.559235 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.559231 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.559212018 +0000 UTC m=+3.975926423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:47.622103 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.622071 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:53:47.660012 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.659966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:47.660163 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:47.660021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:47.660163 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.660103 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:47.660163 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.660115 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:47.660163 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.660161 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.660145757 +0000 UTC m=+4.076860146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:47.660291 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:47.660175 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.660168842 +0000 UTC m=+4.076883276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:48.005688 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.005646 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241032c6_fb35_4a4c_917d_539cc479b8cc.slice/crio-c289bafd62c27637e912d33b32c2ba4cc1347be4e58a8589e72400cee34e4bdf WatchSource:0}: Error finding container c289bafd62c27637e912d33b32c2ba4cc1347be4e58a8589e72400cee34e4bdf: Status 404 returned error can't find the container with id c289bafd62c27637e912d33b32c2ba4cc1347be4e58a8589e72400cee34e4bdf Jul 09 18:53:48.007197 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.007172 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0824ce37_5f83_4f67_bf0f_56c7f723f3d2.slice/crio-19b2a48e1d00f6473d3b3b915770297dc379ff0478e3068c230a928b9255519c WatchSource:0}: Error finding container 19b2a48e1d00f6473d3b3b915770297dc379ff0478e3068c230a928b9255519c: Status 404 returned error can't find the container with id 19b2a48e1d00f6473d3b3b915770297dc379ff0478e3068c230a928b9255519c Jul 09 18:53:48.009164 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.008880 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa94099b_e104_4adb_a96a_e9737f9c043b.slice/crio-6994226d121f9db03628a306074cc9f356641397b9d61fe88537a218230de24a WatchSource:0}: Error finding container 6994226d121f9db03628a306074cc9f356641397b9d61fe88537a218230de24a: Status 404 returned error can't find the container with id 6994226d121f9db03628a306074cc9f356641397b9d61fe88537a218230de24a Jul 09 18:53:48.010841 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.010805 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf59030_65db_468f_ba49_ee906a44d696.slice/crio-48ec0c14907ca94571e330da722c8da55689856e4be2975c7b29437eb4f34757 WatchSource:0}: Error finding container 48ec0c14907ca94571e330da722c8da55689856e4be2975c7b29437eb4f34757: Status 404 returned error can't find the container with id 48ec0c14907ca94571e330da722c8da55689856e4be2975c7b29437eb4f34757 Jul 09 18:53:48.030578 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.030534 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d580e6d_4c52_4ed8_b1e3_f2776b7c5bd7.slice/crio-3777cb64ba2a66b3fba49fd1d96053825ec1031f283382395898458a8877d4d0 WatchSource:0}: Error finding container 3777cb64ba2a66b3fba49fd1d96053825ec1031f283382395898458a8877d4d0: Status 404 returned error can't find the container with id 3777cb64ba2a66b3fba49fd1d96053825ec1031f283382395898458a8877d4d0 Jul 09 18:53:48.031077 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.031029 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9559bce9_1971_40aa_8e5d_de6f89e2d607.slice/crio-3fb128c992571269c4ba3089001ea714973b85619f3ce9518ded11255b8c6a6f WatchSource:0}: Error finding container 3fb128c992571269c4ba3089001ea714973b85619f3ce9518ded11255b8c6a6f: Status 404 returned error can't find the container with id 3fb128c992571269c4ba3089001ea714973b85619f3ce9518ded11255b8c6a6f Jul 09 18:53:48.031773 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.031749 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19f6665_52bb_43cd_8fc8_f3ca82df5cad.slice/crio-c6819e9d10aaf0d080e2e9804854eae917c67f171010338e76ab5cdfd2af6d5b WatchSource:0}: Error finding container c6819e9d10aaf0d080e2e9804854eae917c67f171010338e76ab5cdfd2af6d5b: Status 404 returned error can't find the container with id c6819e9d10aaf0d080e2e9804854eae917c67f171010338e76ab5cdfd2af6d5b Jul 09 18:53:48.032359 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.032328 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:47 +0000 UTC" deadline="2028-03-31 00:03:06.941980266 +0000 UTC" Jul 09 18:53:48.032425 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.032357 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15125h9m18.909627959s" Jul 09 18:53:48.032860 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.032841 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d74512_0c1b_4f1b_b025_8daa776a5c8a.slice/crio-06a091f5c362389459eb169cd11e52ae164e7d485b753233a9283fabb07273b1 WatchSource:0}: Error finding container 06a091f5c362389459eb169cd11e52ae164e7d485b753233a9283fabb07273b1: Status 404 returned error can't find the container with id 06a091f5c362389459eb169cd11e52ae164e7d485b753233a9283fabb07273b1 Jul 09 18:53:48.034044 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:53:48.034025 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be92434_685d_46cd_bc29_b654ee8218e7.slice/crio-e8748739e7e96a5c94f4572e37caf28669f0c484f9b6c6717f054df41b042018 WatchSource:0}: Error finding container e8748739e7e96a5c94f4572e37caf28669f0c484f9b6c6717f054df41b042018: Status 404 returned error can't find the container with id e8748739e7e96a5c94f4572e37caf28669f0c484f9b6c6717f054df41b042018 Jul 09 18:53:48.100698 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.100558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:48.100818 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.100801 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:48.112392 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.112363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n9jq9" event={"ID":"9559bce9-1971-40aa-8e5d-de6f89e2d607","Type":"ContainerStarted","Data":"3fb128c992571269c4ba3089001ea714973b85619f3ce9518ded11255b8c6a6f"} Jul 09 18:53:48.116204 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.116180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5tbn" event={"ID":"e19f6665-52bb-43cd-8fc8-f3ca82df5cad","Type":"ContainerStarted","Data":"c6819e9d10aaf0d080e2e9804854eae917c67f171010338e76ab5cdfd2af6d5b"} Jul 09 18:53:48.117432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.117407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc4s2" event={"ID":"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7","Type":"ContainerStarted","Data":"3777cb64ba2a66b3fba49fd1d96053825ec1031f283382395898458a8877d4d0"} Jul 09 18:53:48.120274 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.120254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-79j55" event={"ID":"daf59030-65db-468f-ba49-ee906a44d696","Type":"ContainerStarted","Data":"48ec0c14907ca94571e330da722c8da55689856e4be2975c7b29437eb4f34757"} Jul 09 18:53:48.126924 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.126903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"19b2a48e1d00f6473d3b3b915770297dc379ff0478e3068c230a928b9255519c"} Jul 09 18:53:48.130663 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.130634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" event={"ID":"aa94099b-e104-4adb-a96a-e9737f9c043b","Type":"ContainerStarted","Data":"6994226d121f9db03628a306074cc9f356641397b9d61fe88537a218230de24a"} Jul 09 18:53:48.132208 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.132182 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerStarted","Data":"c289bafd62c27637e912d33b32c2ba4cc1347be4e58a8589e72400cee34e4bdf"} Jul 09 18:53:48.134281 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.134261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwv46" event={"ID":"6be92434-685d-46cd-bc29-b654ee8218e7","Type":"ContainerStarted","Data":"e8748739e7e96a5c94f4572e37caf28669f0c484f9b6c6717f054df41b042018"} Jul 09 18:53:48.135275 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.135254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w8pjl" event={"ID":"86d74512-0c1b-4f1b-b025-8daa776a5c8a","Type":"ContainerStarted","Data":"06a091f5c362389459eb169cd11e52ae164e7d485b753233a9283fabb07273b1"} Jul 09 18:53:48.565874 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.565781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:48.566042 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.565998 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:48.566042 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.566017 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:48.566042 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.566030 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:48.566199 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.566089 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.566068712 +0000 UTC m=+5.982783106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:48.666562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.666516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:48.666760 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:48.666598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:48.666834 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.666773 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:48.666905 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.666841 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.666821185 +0000 UTC m=+6.083535587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:48.667305 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.667208 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:48.667305 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:48.667259 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.66724331 +0000 UTC m=+6.083957702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:49.033519 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.033384 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:47 +0000 UTC" deadline="2028-01-03 15:07:26.703496564 +0000 UTC" Jul 09 18:53:49.033519 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.033441 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13028h13m37.670059195s" Jul 09 18:53:49.113120 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.113060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:49.113287 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:49.113187 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:49.113580 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.113562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:49.113677 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:49.113657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:49.158216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.158179 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8267dc650a784c1be2605ae2dc268fa" containerID="87990444dbbb398c6bdf847a2b4cbb1ee6d19c5fadfc3190af6cc962b3ce4377" exitCode=0 Jul 09 18:53:49.158396 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.158275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" event={"ID":"a8267dc650a784c1be2605ae2dc268fa","Type":"ContainerDied","Data":"87990444dbbb398c6bdf847a2b4cbb1ee6d19c5fadfc3190af6cc962b3ce4377"} Jul 09 18:53:49.164501 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:49.164298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" event={"ID":"7acd9c4c33c3ec7d300bea1511a846a1","Type":"ContainerStarted","Data":"11c407d70bbec0282aa82591cce758ab40d66756b2ad76d3b79510ee2c695895"} Jul 09 18:53:50.100416 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.100366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:50.100914 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.100528 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:50.186829 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.186794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" event={"ID":"a8267dc650a784c1be2605ae2dc268fa","Type":"ContainerStarted","Data":"fa5c33de1e84a54e572abdba7ad86c95387a295115821ceab382a9b4f1ab595d"} Jul 09 18:53:50.220974 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.220905 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-3.ec2.internal" podStartSLOduration=4.220880728 podStartE2EDuration="4.220880728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:49.196932453 +0000 UTC m=+4.613646864" watchObservedRunningTime="2026-07-09 18:53:50.220880728 +0000 UTC m=+5.637595140" Jul 09 18:53:50.221164 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.221071 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-3.ec2.internal" podStartSLOduration=4.221060351 podStartE2EDuration="4.221060351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:50.219905891 +0000 UTC m=+5.636620302" watchObservedRunningTime="2026-07-09 18:53:50.221060351 +0000 UTC m=+5.637774767" Jul 09 18:53:50.580209 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.580120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:50.580364 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.580311 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:50.580364 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.580332 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:50.580364 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.580346 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:50.580529 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.580409 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.580388296 +0000 UTC m=+9.997102710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:50.680790 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.680752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:50.680959 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:50.680829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:50.681037 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.680988 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:50.681093 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.681054 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.681034675 +0000 UTC m=+10.097749068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:50.681466 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.681446 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:50.681553 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:50.681499 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.681484957 +0000 UTC m=+10.098199350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:51.100859 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:51.100539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:51.100859 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:51.100722 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:51.101419 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:51.101235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:51.101419 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:51.101327 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:52.100292 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:52.100255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:52.100464 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:52.100401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:53.100940 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:53.100821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:53.101365 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:53.100951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:53.101365 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:53.100821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:53.101476 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:53.101395 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:54.100564 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:54.100534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:54.100774 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.100682 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:54.613838 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:54.613800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:54.614270 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.613998 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:53:54.614270 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.614028 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:53:54.614270 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.614042 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:54.614270 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.614105 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.614086884 +0000 UTC m=+18.030801284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:54.714716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:54.714798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.714855 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.714933 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.714911651 +0000 UTC m=+18.131626055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.714939 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:54.715060 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:54.715019 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.715004722 +0000 UTC m=+18.131719113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:53:55.100968 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:55.100920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:55.101147 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:55.101028 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:55.101147 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:55.101050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:55.101257 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:55.101144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:56.100349 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:56.100315 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:56.100763 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:56.100428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:57.101008 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:57.100916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:57.101008 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:57.100975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:57.101566 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:57.101074 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:53:57.101566 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:57.101326 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:58.100912 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:58.100387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:53:58.100912 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:58.100544 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:53:59.100203 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:59.100177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:53:59.100352 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:59.100300 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:53:59.100412 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:53:59.100368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:53:59.100476 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:53:59.100458 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:00.100776 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:00.100747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:00.101210 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:00.100860 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:01.100343 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:01.100308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:01.100525 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:01.100418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:01.100525 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:01.100478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:01.100631 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:01.100567 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:02.100175 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:02.100130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:02.100639 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.100283 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:02.680822 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:02.680785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:02.681081 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.680938 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:54:02.681081 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.680958 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:54:02.681081 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.680971 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:02.681081 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.681036 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.681018225 +0000 UTC m=+34.097732625 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:02.781355 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:02.781320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:02.781538 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:02.781395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:02.781538 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.781492 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:02.781538 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.781528 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:02.781695 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.781562 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.781540424 +0000 UTC m=+34.198254811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:02.781695 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:02.781593 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.781574549 +0000 UTC m=+34.198288938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:03.100486 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:03.100459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:03.100911 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:03.100502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:03.100911 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:03.100576 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:03.100911 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:03.100701 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:04.100777 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:04.100746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:04.101231 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:04.100866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:05.101917 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:05.101882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:05.102348 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:05.101972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:05.102348 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:05.101996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:05.102348 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:05.102098 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:06.100223 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:06.100192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:06.100408 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:06.100325 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:07.100225 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.100067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:07.100898 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.100121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:07.100898 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:07.100314 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:07.100898 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:07.100376 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:07.226035 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.225995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" event={"ID":"aa94099b-e104-4adb-a96a-e9737f9c043b","Type":"ContainerStarted","Data":"a603c492adea7cc6171627c7d64f84a004fb822c39e0372217affa236d552fbd"} Jul 09 18:54:07.227591 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.227569 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="ed79c9fa4ed6b3f3d143eaa372da4e321d6f2a6c478637eecef9a1e42c70668d" exitCode=0 Jul 09 18:54:07.227693 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.227638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"ed79c9fa4ed6b3f3d143eaa372da4e321d6f2a6c478637eecef9a1e42c70668d"} Jul 09 18:54:07.229265 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.229244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dwv46" event={"ID":"6be92434-685d-46cd-bc29-b654ee8218e7","Type":"ContainerStarted","Data":"209939b4f9c4ad08f0a10421c9fc8f9aab5ab9a90b3c103e2793ef42733128fb"} Jul 09 18:54:07.230700 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.230678 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n9jq9" event={"ID":"9559bce9-1971-40aa-8e5d-de6f89e2d607","Type":"ContainerStarted","Data":"24e5091ca703d367e82d4e75bab1534c333317f1ab00cc4adb97036ff0ca17f4"} Jul 09 18:54:07.232201 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.232154 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5tbn" event={"ID":"e19f6665-52bb-43cd-8fc8-f3ca82df5cad","Type":"ContainerStarted","Data":"2f2106d441b821d293d286afcc6618c4fceb2c2da15177884da93188b6ba5389"} Jul 09 18:54:07.233558 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.233537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc4s2" event={"ID":"0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7","Type":"ContainerStarted","Data":"df764fc814afdf5c39828904dd6ffae701a31e455efa0e3895f0ff7f97ab807d"} Jul 09 18:54:07.235019 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.234997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-79j55" event={"ID":"daf59030-65db-468f-ba49-ee906a44d696","Type":"ContainerStarted","Data":"89ce5179db0423ed806a43a827e595bfa2924ecefb52e09955cd1ba45817302f"} Jul 09 18:54:07.237110 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237094 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:54:07.237351 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237333 2578 generic.go:356] "Generic (PLEG): container finished" podID="0824ce37-5f83-4f67-bf0f-56c7f723f3d2" containerID="cf2b107b7e296413dd92dc6869ef907d012b76080bdc88f38911e8a19d46bc7c" exitCode=1 Jul 09 18:54:07.237407 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"0039b37e941006f65d8ecf7d27ed0404697468aa30b5b1558ec3c9dceee778fe"} Jul 09 18:54:07.237445 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"12e03bbe48c5b83b450a6947742f1bdaebf532a02a96aea83449f7a3e697a984"} Jul 09 18:54:07.237477 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerDied","Data":"cf2b107b7e296413dd92dc6869ef907d012b76080bdc88f38911e8a19d46bc7c"} Jul 09 18:54:07.237507 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.237477 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"c72de87340b98fc0bbd351fd30e14cd1677f98050f80ca384429ebd9e69f477c"} Jul 09 18:54:07.337105 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.336918 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-79j55" podStartSLOduration=11.632257615 podStartE2EDuration="22.336900225s" totalImagesPullingTime="10.70464261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:07.336421624 +0000 UTC m=+22.753136034" watchObservedRunningTime="2026-07-09 18:54:07.336900225 +0000 UTC m=+22.753614633" Jul 09 18:54:07.373052 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.373008 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pc4s2" podStartSLOduration=11.715049298 podStartE2EDuration="22.372992025s" totalImagesPullingTime="10.657942727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:07.372456318 +0000 UTC m=+22.789170729" watchObservedRunningTime="2026-07-09 18:54:07.372992025 +0000 UTC m=+22.789706433" Jul 09 18:54:07.396125 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.396075 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dwv46" podStartSLOduration=3.751218495 podStartE2EDuration="22.396055651s" totalImagesPullingTime="18.644837156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:07.395768354 +0000 UTC m=+22.812482777" watchObservedRunningTime="2026-07-09 18:54:07.396055651 +0000 UTC m=+22.812770063" Jul 09 18:54:07.414346 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.414293 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n9jq9" podStartSLOduration=11.718422066 podStartE2EDuration="22.414273077s" totalImagesPullingTime="10.695851011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:07.414199263 +0000 UTC m=+22.830913685" watchObservedRunningTime="2026-07-09 18:54:07.414273077 +0000 UTC m=+22.830987488" Jul 09 18:54:07.441781 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:07.441713 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l5tbn" podStartSLOduration=3.743359921 podStartE2EDuration="22.441693205s" totalImagesPullingTime="18.698333284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:07.441210522 +0000 UTC m=+22.857924930" watchObservedRunningTime="2026-07-09 18:54:07.441693205 +0000 UTC m=+22.858407616" Jul 09 18:54:08.100797 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.100756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:08.101368 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:08.100886 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:08.243386 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.243337 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:54:08.243860 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.243829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"ae6f191062ad6366b7fcf9799b062e7adc3e6e316c4800940306c2dd27116e88"} Jul 09 18:54:08.243961 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.243902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"3dfbcdd85742b7aa32760ec1663cae68df8fbeedf2855af2ad67c21dc1fbb73e"} Jul 09 18:54:08.245856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.245809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w8pjl" event={"ID":"86d74512-0c1b-4f1b-b025-8daa776a5c8a","Type":"ContainerStarted","Data":"315ed91b39f9f30e814361b6ed0d196a9a8a4b2e29a59d67847cf18635a95445"} Jul 09 18:54:08.265316 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.265261 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w8pjl" podStartSLOduration=12.574478192 podStartE2EDuration="23.265240881s" totalImagesPullingTime="10.690762689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:08.264804741 +0000 UTC m=+23.681519149" watchObservedRunningTime="2026-07-09 18:54:08.265240881 +0000 UTC m=+23.681955297" Jul 09 18:54:08.287874 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:08.287849 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 18:54:09.002302 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.002199 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:54:08.287867503Z","UUID":"898b60a8-caaf-4f99-b06e-e601b238e0be","Handler":null,"Name":"","Endpoint":""} Jul 09 18:54:09.003652 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.003631 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 18:54:09.003805 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.003677 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 18:54:09.100628 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.100592 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:09.100896 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:09.100762 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:09.100896 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.100603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:09.101361 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:09.100963 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:09.249441 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.249406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" event={"ID":"aa94099b-e104-4adb-a96a-e9737f9c043b","Type":"ContainerStarted","Data":"a415bd8efcae0110de02466905245bfac4a9d431f68205943cb2a1b36919262b"} Jul 09 18:54:09.371405 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.371313 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:54:09.372224 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:09.372202 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:54:10.100349 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.100189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:10.100501 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:10.100457 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:10.254931 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.254859 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:54:10.255313 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.255232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"9822387c4151009b3ebf07ed9ab215d8da37d2aea6bfb55db1084314e188ceec"} Jul 09 18:54:10.257211 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.257189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" event={"ID":"aa94099b-e104-4adb-a96a-e9737f9c043b","Type":"ContainerStarted","Data":"ad50d935c2bbeb43e668251208fc7370154a6ef0895c9e39c4b9a329e731ad77"} Jul 09 18:54:10.257482 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.257461 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:54:10.257878 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.257863 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pc4s2" Jul 09 18:54:10.295692 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:10.295646 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2pcb" podStartSLOduration=11.785148733 podStartE2EDuration="25.295632661s" totalImagesPullingTime="13.510483928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:10.278647062 +0000 UTC m=+25.695361476" watchObservedRunningTime="2026-07-09 18:54:10.295632661 +0000 UTC m=+25.712347071" Jul 09 18:54:11.100890 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:11.100808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:11.101114 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:11.100925 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:11.101114 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:11.100979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:11.101114 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:11.101071 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:12.100360 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:12.100273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:12.100968 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:12.100401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:13.100415 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.100231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:13.100991 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.100256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:13.100991 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:13.100494 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:13.100991 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:13.100556 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:13.266889 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.266862 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:54:13.267299 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.267266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"5fc93055ba041a6ebf997fe4ed9de41a1d830ca1246b3be0e1f7ead74fcfda97"} Jul 09 18:54:13.267572 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.267534 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:13.267700 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.267605 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:13.267700 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.267653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:13.267700 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.267671 2578 scope.go:117] "RemoveContainer" containerID="cf2b107b7e296413dd92dc6869ef907d012b76080bdc88f38911e8a19d46bc7c" Jul 09 18:54:13.269466 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.269438 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="5c022ed0d9897a3974860910dfb83949f9a5e0b1d584bb848e20968172c671c0" exitCode=0 Jul 09 18:54:13.269575 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.269484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"5c022ed0d9897a3974860910dfb83949f9a5e0b1d584bb848e20968172c671c0"} Jul 09 18:54:13.286008 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.285986 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:13.286095 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:13.286088 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:14.100343 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:14.100311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:14.100554 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:14.100435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:14.275784 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:14.275758 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:54:14.276104 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:14.276083 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" event={"ID":"0824ce37-5f83-4f67-bf0f-56c7f723f3d2","Type":"ContainerStarted","Data":"b4ea3b9871e6fc994446150838c421149bee5661ab3c9c52964151fef49a6703"} Jul 09 18:54:15.101330 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:15.101291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:15.101835 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:15.101411 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:15.101835 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:15.101457 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:15.101835 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:15.101580 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:15.280852 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:15.280819 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="be8feb39f41ace6656951408bd29aa10f811b049fbb17537f957d6612566023b" exitCode=0 Jul 09 18:54:15.281004 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:15.280913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"be8feb39f41ace6656951408bd29aa10f811b049fbb17537f957d6612566023b"} Jul 09 18:54:15.309023 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:15.308957 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" podStartSLOduration=11.556094881 podStartE2EDuration="30.308941647s" totalImagesPullingTime="18.752846766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:14.352714606 +0000 UTC m=+29.769429015" watchObservedRunningTime="2026-07-09 18:54:15.308941647 +0000 UTC m=+30.725656056" Jul 09 18:54:16.100218 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:16.100180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:16.100392 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:16.100331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:17.100198 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:17.100165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:17.100558 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:17.100280 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:17.100558 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:17.100331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:17.100558 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:17.100426 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:17.289526 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:17.289492 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="1c9bc026e0f23dbc2f63d2a58bad51e7ab5e5369dab189b3bc96bcf55d4fd7d6" exitCode=0 Jul 09 18:54:17.289526 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:17.289529 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"1c9bc026e0f23dbc2f63d2a58bad51e7ab5e5369dab189b3bc96bcf55d4fd7d6"} Jul 09 18:54:18.099932 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:18.099901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:18.100094 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.100030 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:18.691207 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:18.691173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:18.691655 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.691354 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:54:18.691655 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.691378 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:54:18.691655 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.691398 2578 projected.go:196] Error preparing data for projected volume kube-api-access-79wxz for pod openshift-network-diagnostics/network-check-target-7hvxv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:18.691655 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.691457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz podName:0a434b2e-430b-4531-a9c5-15172d73e62a nodeName:}" failed. No retries permitted until 2026-07-09 18:54:50.691443033 +0000 UTC m=+66.108157421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-79wxz" (UniqueName: "kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz") pod "network-check-target-7hvxv" (UID: "0a434b2e-430b-4531-a9c5-15172d73e62a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:54:18.791936 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:18.791896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:18.792119 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:18.791962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:18.792119 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.792003 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:18.792119 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.792066 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:18.792119 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.792072 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs podName:f2f1bca4-a35c-40c0-8591-b4a7f233e6b8 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:50.792053716 +0000 UTC m=+66.208768109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs") pod "network-metrics-daemon-f549x" (UID: "f2f1bca4-a35c-40c0-8591-b4a7f233e6b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:54:18.792119 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:18.792115 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret podName:2c4b166f-98c3-43d4-a8b6-6872182419f6 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:50.792098951 +0000 UTC m=+66.208813345 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret") pod "global-pull-secret-syncer-g2nbg" (UID: "2c4b166f-98c3-43d4-a8b6-6872182419f6") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:54:19.100641 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:19.100385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:19.100839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:19.100395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:19.100839 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:19.100697 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:19.100839 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:19.100775 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:20.100818 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:20.100788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:20.101199 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:20.100913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:21.100267 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:21.100229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:21.100470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:21.100270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:21.100470 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:21.100351 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:21.100579 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:21.100488 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:22.100450 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:22.100412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:22.100925 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:22.100548 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:23.100148 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:23.100053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:23.100148 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:23.100090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:23.100322 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:23.100196 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:23.100322 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:23.100282 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:23.306483 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:23.306451 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="b12a092057a9dcd6f22753045e2be764022174aa1003ed14ae02598ba9e5c2fc" exitCode=0 Jul 09 18:54:23.306876 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:23.306508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"b12a092057a9dcd6f22753045e2be764022174aa1003ed14ae02598ba9e5c2fc"} Jul 09 18:54:24.100109 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:24.100067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:24.100290 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:24.100183 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:24.311470 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:24.311430 2578 generic.go:356] "Generic (PLEG): container finished" podID="241032c6-fb35-4a4c-917d-539cc479b8cc" containerID="2f5b109007ab12e44613153e22e61c5d7c961f09e7ce1589752f517166a9b069" exitCode=0 Jul 09 18:54:24.311836 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:24.311500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerDied","Data":"2f5b109007ab12e44613153e22e61c5d7c961f09e7ce1589752f517166a9b069"} Jul 09 18:54:25.101249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:25.101215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:25.101426 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:25.101303 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:25.101426 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:25.101402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:25.101538 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:25.101520 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:25.316841 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:25.316806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" event={"ID":"241032c6-fb35-4a4c-917d-539cc479b8cc","Type":"ContainerStarted","Data":"b6baca81a32c1c2dc2a5800c28b0816b6ae27df7ca38f382457674de824ebc5c"} Jul 09 18:54:25.342346 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:25.342298 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sdfpx" podStartSLOduration=16.031092676 podStartE2EDuration="40.342282143s" totalImagesPullingTime="24.311189467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:25.342189683 +0000 UTC m=+40.758904102" watchObservedRunningTime="2026-07-09 18:54:25.342282143 +0000 UTC m=+40.758996553" Jul 09 18:54:26.100013 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:26.099982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:26.100198 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:26.100087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:27.100748 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:27.100695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:27.101210 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:27.100849 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:27.101210 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:27.100891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:27.101210 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:27.100979 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:28.100169 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:28.100132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:28.100347 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:28.100247 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:29.099938 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:29.099901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:29.100395 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:29.099901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:29.100395 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:29.100009 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:29.100395 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:29.100114 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:30.100798 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:30.100763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:30.101283 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:30.100874 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:31.100183 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:31.100153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:31.100347 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:31.100166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:31.100347 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:31.100292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:31.100347 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:31.100327 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:32.100539 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:32.100507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:32.100938 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:32.100637 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:33.100750 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:33.100700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:33.100750 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:33.100723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:33.101220 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:33.100842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:33.101220 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:33.100924 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:34.100047 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:34.100017 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:34.100212 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:34.100124 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:35.101700 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.101661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:35.102393 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:35.101815 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:35.102393 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.101872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:35.102393 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:35.101985 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:35.442129 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.442089 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g2nbg"] Jul 09 18:54:35.442316 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.442188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:35.442316 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:35.442277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:35.445287 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.445208 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hvxv"] Jul 09 18:54:35.445446 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.445320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:35.445446 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:35.445425 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:35.445915 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.445889 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f549x"] Jul 09 18:54:35.446038 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:35.446001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:35.446130 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:35.446111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:37.099995 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:37.099953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:37.099995 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:37.099990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:37.100445 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:37.100087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:37.100445 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:37.100123 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:37.100445 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:37.100166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:37.100445 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:37.100248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:39.100549 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.100510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:39.100964 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.100521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:39.100964 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:39.100611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hvxv" podUID="0a434b2e-430b-4531-a9c5-15172d73e62a" Jul 09 18:54:39.100964 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.100628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:39.100964 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:39.100743 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f549x" podUID="f2f1bca4-a35c-40c0-8591-b4a7f233e6b8" Jul 09 18:54:39.100964 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:54:39.100803 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g2nbg" podUID="2c4b166f-98c3-43d4-a8b6-6872182419f6" Jul 09 18:54:39.398931 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.398903 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-3.ec2.internal" event="NodeReady" Jul 09 18:54:39.399085 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.399072 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 18:54:39.457275 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.457237 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ctn98"] Jul 09 18:54:39.467008 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.466979 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8v9xg"] Jul 09 18:54:39.467178 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.467107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.471197 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.471177 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 18:54:39.471333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.471283 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xvss5\"" Jul 09 18:54:39.471333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.471295 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 18:54:39.479155 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.479133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ctn98"] Jul 09 18:54:39.479265 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.479167 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8v9xg"] Jul 09 18:54:39.479265 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.479247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.485245 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.485064 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 18:54:39.485245 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.485084 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 18:54:39.485245 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.485222 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 18:54:39.485428 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.485321 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 18:54:39.485428 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.485350 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2d2w\"" Jul 09 18:54:39.487828 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.487808 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r5g"] Jul 09 18:54:39.501961 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.501941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.506683 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.506659 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r5g"] Jul 09 18:54:39.507118 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.506922 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 18:54:39.507118 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.506928 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 18:54:39.507118 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.506978 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 18:54:39.507329 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.507305 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl7rk\"" Jul 09 18:54:39.546924 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.546896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10d06887-6db9-405b-929f-fceb9b35ad05-metrics-tls\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.546924 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.546925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-kube-api-access-rkmc5\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.547094 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.546944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d06887-6db9-405b-929f-fceb9b35ad05-tmp-dir\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.547094 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.546975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.547094 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4f64da-6aee-4c44-835b-bbfd5a80e428-data-volume\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.547094 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10d06887-6db9-405b-929f-fceb9b35ad05-config-volume\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.547094 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9266\" (UniqueName: \"kubernetes.io/projected/10d06887-6db9-405b-929f-fceb9b35ad05-kube-api-access-r9266\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.547243 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4f64da-6aee-4c44-835b-bbfd5a80e428-crio-socket\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.547243 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvm6b\" (UniqueName: \"kubernetes.io/projected/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-api-access-jvm6b\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.547243 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4f64da-6aee-4c44-835b-bbfd5a80e428-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.547243 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.547202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-cert\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.648214 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9266\" (UniqueName: \"kubernetes.io/projected/10d06887-6db9-405b-929f-fceb9b35ad05-kube-api-access-r9266\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.648214 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4f64da-6aee-4c44-835b-bbfd5a80e428-crio-socket\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648378 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm6b\" (UniqueName: \"kubernetes.io/projected/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-api-access-jvm6b\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648378 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4f64da-6aee-4c44-835b-bbfd5a80e428-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648378 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-cert\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.648378 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10d06887-6db9-405b-929f-fceb9b35ad05-metrics-tls\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.648553 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-kube-api-access-rkmc5\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.648553 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d06887-6db9-405b-929f-fceb9b35ad05-tmp-dir\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.648553 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4f64da-6aee-4c44-835b-bbfd5a80e428-crio-socket\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648553 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648553 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4f64da-6aee-4c44-835b-bbfd5a80e428-data-volume\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.648803 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10d06887-6db9-405b-929f-fceb9b35ad05-config-volume\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.648886 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.648866 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10d06887-6db9-405b-929f-fceb9b35ad05-tmp-dir\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.649185 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.649126 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4f64da-6aee-4c44-835b-bbfd5a80e428-data-volume\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.649276 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.649243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.649320 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.649272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10d06887-6db9-405b-929f-fceb9b35ad05-config-volume\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.652428 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.652403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4f64da-6aee-4c44-835b-bbfd5a80e428-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.652522 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.652448 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10d06887-6db9-405b-929f-fceb9b35ad05-metrics-tls\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.652590 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.652574 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-cert\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.667366 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.667346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm6b\" (UniqueName: \"kubernetes.io/projected/0c4f64da-6aee-4c44-835b-bbfd5a80e428-kube-api-access-jvm6b\") pod \"insights-runtime-extractor-8v9xg\" (UID: \"0c4f64da-6aee-4c44-835b-bbfd5a80e428\") " pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.667481 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.667447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9266\" (UniqueName: \"kubernetes.io/projected/10d06887-6db9-405b-929f-fceb9b35ad05-kube-api-access-r9266\") pod \"dns-default-ctn98\" (UID: \"10d06887-6db9-405b-929f-fceb9b35ad05\") " pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.667481 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.667468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb-kube-api-access-rkmc5\") pod \"ingress-canary-x9r5g\" (UID: \"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb\") " pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.776650 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.776618 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:39.787775 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.787752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8v9xg" Jul 09 18:54:39.811163 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.811142 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9r5g" Jul 09 18:54:39.980307 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.980269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ctn98"] Jul 09 18:54:39.983321 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.983160 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9r5g"] Jul 09 18:54:39.984178 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:39.984046 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8v9xg"] Jul 09 18:54:39.989108 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:39.987814 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d06887_6db9_405b_929f_fceb9b35ad05.slice/crio-9ad7a0ffa94cbabd8250d1e6a4fe9203ecc6e2929a140f4c4f93775cbf962052 WatchSource:0}: Error finding container 9ad7a0ffa94cbabd8250d1e6a4fe9203ecc6e2929a140f4c4f93775cbf962052: Status 404 returned error can't find the container with id 9ad7a0ffa94cbabd8250d1e6a4fe9203ecc6e2929a140f4c4f93775cbf962052 Jul 09 18:54:40.349467 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:40.349369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9r5g" event={"ID":"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb","Type":"ContainerStarted","Data":"5402023ac307821ac8230012164ccdbc371484d71c411060a34e8cee3c9612f5"} Jul 09 18:54:40.350379 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:40.350358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8v9xg" event={"ID":"0c4f64da-6aee-4c44-835b-bbfd5a80e428","Type":"ContainerStarted","Data":"8d469fb6c66324e2f7e5633b804bcff690797ca5cf4a4db883bcc1ff719c422b"} Jul 09 18:54:40.351389 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:40.351360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ctn98" event={"ID":"10d06887-6db9-405b-929f-fceb9b35ad05","Type":"ContainerStarted","Data":"9ad7a0ffa94cbabd8250d1e6a4fe9203ecc6e2929a140f4c4f93775cbf962052"} Jul 09 18:54:41.102385 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.100978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:41.102385 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.100985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:41.102385 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.101091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.108304 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkpmb\"" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.108567 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47bt6\"" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.108850 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.109049 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.109273 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:54:41.110625 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.109273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:54:41.256896 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.256858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:54:41.259074 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.259050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.263621 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.262398 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 18:54:41.263621 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.262599 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 18:54:41.263621 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.262892 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 18:54:41.263621 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.263419 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 18:54:41.263621 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.263577 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cplfs\"" Jul 09 18:54:41.263973 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.263822 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 18:54:41.264050 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.264024 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 18:54:41.264321 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.264220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 18:54:41.270297 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.270265 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 18:54:41.270418 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.270356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:54:41.364199 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364199 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364199 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364658 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364658 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364658 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vp5f\" (UniqueName: \"kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.364658 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.364284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465630 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465630 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vp5f\" (UniqueName: \"kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465851 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465851 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465851 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465851 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.465968 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.465865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.467797 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.466706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.467797 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.467236 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.468210 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.468152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.468460 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.468439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.472035 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.472013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.472444 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.472414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.479180 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.479157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vp5f\" (UniqueName: \"kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f\") pod \"console-6b8868bdb9-m8tqk\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:41.574191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:41.574158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:45.075481 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.074591 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:54:45.081716 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:45.080444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8f78ce_0498_4786_8af3_0d34a883eff5.slice/crio-5e5941d1dc680f25da9ecad553f55ab0548671ca2a95d7233c57958b69159ec9 WatchSource:0}: Error finding container 5e5941d1dc680f25da9ecad553f55ab0548671ca2a95d7233c57958b69159ec9: Status 404 returned error can't find the container with id 5e5941d1dc680f25da9ecad553f55ab0548671ca2a95d7233c57958b69159ec9 Jul 09 18:54:45.291241 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.291164 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4hc8" Jul 09 18:54:45.365616 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.365571 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8868bdb9-m8tqk" event={"ID":"9d8f78ce-0498-4786-8af3-0d34a883eff5","Type":"ContainerStarted","Data":"5e5941d1dc680f25da9ecad553f55ab0548671ca2a95d7233c57958b69159ec9"} Jul 09 18:54:45.366999 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.366976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8v9xg" event={"ID":"0c4f64da-6aee-4c44-835b-bbfd5a80e428","Type":"ContainerStarted","Data":"be8e40dfdf6d44e9d1b139176f506aac220370e1796d49c96e58195553b6076a"} Jul 09 18:54:45.368569 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.368548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ctn98" event={"ID":"10d06887-6db9-405b-929f-fceb9b35ad05","Type":"ContainerStarted","Data":"bb7fa9f87a02b9dd80d07887923d829592e0eb085aa38b1acfa89821499053b2"} Jul 09 18:54:45.368660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.368592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ctn98" event={"ID":"10d06887-6db9-405b-929f-fceb9b35ad05","Type":"ContainerStarted","Data":"a919297754486d998b05f9ef0819c79daacec98e87ca7b8e373e2aaa3856287d"} Jul 09 18:54:45.368660 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.368642 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:45.369986 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.369964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9r5g" event={"ID":"16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb","Type":"ContainerStarted","Data":"03b26812584d5d8454af88102e905bb681384c2aecb22546d670d3f33871866e"} Jul 09 18:54:45.389934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.389888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ctn98" podStartSLOduration=1.775689815 podStartE2EDuration="6.389874391s" totalImagesPullingTime="4.614184576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:45.389186588 +0000 UTC m=+60.805900998" watchObservedRunningTime="2026-07-09 18:54:45.389874391 +0000 UTC m=+60.806588801" Jul 09 18:54:45.416049 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:45.415993 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9r5g" podStartSLOduration=1.477954374 podStartE2EDuration="6.415977296s" totalImagesPullingTime="4.938022922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:45.415623666 +0000 UTC m=+60.832338077" watchObservedRunningTime="2026-07-09 18:54:45.415977296 +0000 UTC m=+60.832691697" Jul 09 18:54:46.376476 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.376387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8v9xg" event={"ID":"0c4f64da-6aee-4c44-835b-bbfd5a80e428","Type":"ContainerStarted","Data":"5cda3ab5e07bafc2ce0ebd2c9d299992ad8c098070b11f40ec7f395f2efbd18a"} Jul 09 18:54:46.942143 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.942104 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q"] Jul 09 18:54:46.963862 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.963832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:46.967040 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.966852 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 18:54:46.967040 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.966891 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kjmqv\"" Jul 09 18:54:46.968649 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.968235 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 18:54:46.968649 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.968261 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 18:54:46.968649 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.968300 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 18:54:46.968892 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.968717 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 18:54:46.970780 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.970759 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q"] Jul 09 18:54:46.987890 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:46.987868 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8h8zb"] Jul 09 18:54:47.003768 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.003745 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.008492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.008466 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 18:54:47.008681 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.008663 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 18:54:47.008877 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.008858 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vtbj\"" Jul 09 18:54:47.012898 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.012879 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 18:54:47.014618 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.014595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.014705 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.014665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.014705 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.014699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.014813 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.014746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcztl\" (UniqueName: \"kubernetes.io/projected/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-kube-api-access-zcztl\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.115915 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.115883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-tls\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116100 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.115921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-metrics-client-ca\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116100 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.115999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-sys\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116100 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.116255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.116255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-textfile\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-root\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcztl\" (UniqueName: \"kubernetes.io/projected/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-kube-api-access-zcztl\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.116517 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.116517 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw22v\" (UniqueName: \"kubernetes.io/projected/37952e3a-9fb9-4860-b3c6-d02429d75b58-kube-api-access-kw22v\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116517 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.116517 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-wtmp\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.117011 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.116987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.119149 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.119123 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.119365 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.119332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.127643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.127595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcztl\" (UniqueName: \"kubernetes.io/projected/12911dea-c8cc-4a03-8ca7-7bfd2f269a1e-kube-api-access-zcztl\") pod \"openshift-state-metrics-84bd68b896-fzw6q\" (UID: \"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.217062 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.216967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-tls\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217062 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-metrics-client-ca\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217062 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-sys\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-textfile\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-root\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw22v\" (UniqueName: \"kubernetes.io/projected/37952e3a-9fb9-4860-b3c6-d02429d75b58-kube-api-access-kw22v\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217387 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-wtmp\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217668 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-root\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217668 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217519 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-wtmp\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.217964 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.217829 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37952e3a-9fb9-4860-b3c6-d02429d75b58-sys\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.218099 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.218076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.218176 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.218090 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-textfile\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.218176 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.218086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37952e3a-9fb9-4860-b3c6-d02429d75b58-metrics-client-ca\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.220682 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.220637 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.220809 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.220786 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/37952e3a-9fb9-4860-b3c6-d02429d75b58-node-exporter-tls\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.228419 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.228389 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw22v\" (UniqueName: \"kubernetes.io/projected/37952e3a-9fb9-4860-b3c6-d02429d75b58-kube-api-access-kw22v\") pod \"node-exporter-8h8zb\" (UID: \"37952e3a-9fb9-4860-b3c6-d02429d75b58\") " pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:47.275893 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.275854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" Jul 09 18:54:47.313810 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:47.313780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h8zb" Jul 09 18:54:49.205694 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:49.205650 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37952e3a_9fb9_4860_b3c6_d02429d75b58.slice/crio-163bd82de1753b5d837156f7037cc3114477f245d07cb2d261f1fd200bdcf7bb WatchSource:0}: Error finding container 163bd82de1753b5d837156f7037cc3114477f245d07cb2d261f1fd200bdcf7bb: Status 404 returned error can't find the container with id 163bd82de1753b5d837156f7037cc3114477f245d07cb2d261f1fd200bdcf7bb Jul 09 18:54:49.336854 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.336823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q"] Jul 09 18:54:49.339402 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:49.339365 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12911dea_c8cc_4a03_8ca7_7bfd2f269a1e.slice/crio-0c2facb21a8e5fe2d51b251b47a25e45ed3f86e3408c315969a80fceacf50bc5 WatchSource:0}: Error finding container 0c2facb21a8e5fe2d51b251b47a25e45ed3f86e3408c315969a80fceacf50bc5: Status 404 returned error can't find the container with id 0c2facb21a8e5fe2d51b251b47a25e45ed3f86e3408c315969a80fceacf50bc5 Jul 09 18:54:49.388605 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.388575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h8zb" event={"ID":"37952e3a-9fb9-4860-b3c6-d02429d75b58","Type":"ContainerStarted","Data":"163bd82de1753b5d837156f7037cc3114477f245d07cb2d261f1fd200bdcf7bb"} Jul 09 18:54:49.390081 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.390051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" event={"ID":"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e","Type":"ContainerStarted","Data":"0c2facb21a8e5fe2d51b251b47a25e45ed3f86e3408c315969a80fceacf50bc5"} Jul 09 18:54:49.391797 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.391766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8868bdb9-m8tqk" event={"ID":"9d8f78ce-0498-4786-8af3-0d34a883eff5","Type":"ContainerStarted","Data":"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc"} Jul 09 18:54:49.393952 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.393928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8v9xg" event={"ID":"0c4f64da-6aee-4c44-835b-bbfd5a80e428","Type":"ContainerStarted","Data":"33afb85c9f29f5819146ebb4596ce7682b41aebdd974a9e1df68a63499c681e9"} Jul 09 18:54:49.415847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.415804 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8868bdb9-m8tqk" podStartSLOduration=4.291541498 podStartE2EDuration="8.415788942s" totalImagesPullingTime="4.124247444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:49.414970296 +0000 UTC m=+64.831684707" watchObservedRunningTime="2026-07-09 18:54:49.415788942 +0000 UTC m=+64.832503409" Jul 09 18:54:49.437267 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:49.437212 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8v9xg" podStartSLOduration=1.366016143 podStartE2EDuration="10.437192626s" totalImagesPullingTime="9.071176483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:54:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:49.43699776 +0000 UTC m=+64.853712181" watchObservedRunningTime="2026-07-09 18:54:49.437192626 +0000 UTC m=+64.853907060" Jul 09 18:54:50.043646 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.043611 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-799b685df8-sxvgk"] Jul 09 18:54:50.046200 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.046172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.050758 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.050718 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 18:54:50.050864 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.050835 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2hh9k\"" Jul 09 18:54:50.051780 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.051465 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 18:54:50.051780 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.051762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2e2efdp0t5me\"" Jul 09 18:54:50.052443 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.052033 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 18:54:50.052443 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.052203 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 18:54:50.052443 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.052371 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 18:54:50.066569 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.066546 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-799b685df8-sxvgk"] Jul 09 18:54:50.143697 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa81b2e5-bab7-468b-b239-414689dad316-metrics-client-ca\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.143888 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.143888 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-grpc-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.143888 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.144059 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.144059 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.143959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.144059 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.144011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9zx\" (UniqueName: \"kubernetes.io/projected/fa81b2e5-bab7-468b-b239-414689dad316-kube-api-access-nk9zx\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.144059 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.144042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245003 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.244966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9zx\" (UniqueName: \"kubernetes.io/projected/fa81b2e5-bab7-468b-b239-414689dad316-kube-api-access-nk9zx\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa81b2e5-bab7-468b-b239-414689dad316-metrics-client-ca\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.245480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.245411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-grpc-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.246317 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.246252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa81b2e5-bab7-468b-b239-414689dad316-metrics-client-ca\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248372 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248372 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248372 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248592 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248672 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248652 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.248899 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.248882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa81b2e5-bab7-468b-b239-414689dad316-secret-grpc-tls\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.253945 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.253927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9zx\" (UniqueName: \"kubernetes.io/projected/fa81b2e5-bab7-468b-b239-414689dad316-kube-api-access-nk9zx\") pod \"thanos-querier-799b685df8-sxvgk\" (UID: \"fa81b2e5-bab7-468b-b239-414689dad316\") " pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.358148 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.358125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:54:50.400502 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.400464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h8zb" event={"ID":"37952e3a-9fb9-4860-b3c6-d02429d75b58","Type":"ContainerStarted","Data":"ddd2c131b23b3f9edd890130a06b04f1af424f1213d6eac0f897fe4acfea9e5d"} Jul 09 18:54:50.402541 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.402497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" event={"ID":"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e","Type":"ContainerStarted","Data":"86571b610f66bfbfacae55fc8fcac28dfd2493a13bd53b3d34315f23370a5815"} Jul 09 18:54:50.402667 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.402563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" event={"ID":"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e","Type":"ContainerStarted","Data":"1076735f7ee4a7066191e4d18684fd8200cc63b4c17a9bc6d6c87e3ac51f5a32"} Jul 09 18:54:50.495525 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.495491 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-799b685df8-sxvgk"] Jul 09 18:54:50.499230 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:50.499204 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa81b2e5_bab7_468b_b239_414689dad316.slice/crio-8aa113a981efe897f5d6f866c5f67483556ee0a57425d487cff795013808beb0 WatchSource:0}: Error finding container 8aa113a981efe897f5d6f866c5f67483556ee0a57425d487cff795013808beb0: Status 404 returned error can't find the container with id 8aa113a981efe897f5d6f866c5f67483556ee0a57425d487cff795013808beb0 Jul 09 18:54:50.750215 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.750175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:50.753379 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.753357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:54:50.764082 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.764059 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:54:50.774520 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.774494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wxz\" (UniqueName: \"kubernetes.io/projected/0a434b2e-430b-4531-a9c5-15172d73e62a-kube-api-access-79wxz\") pod \"network-check-target-7hvxv\" (UID: \"0a434b2e-430b-4531-a9c5-15172d73e62a\") " pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:50.851162 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.851124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:50.851317 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.851180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:50.855150 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.855122 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:54:50.855290 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.855227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:54:50.864530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.864497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2f1bca4-a35c-40c0-8591-b4a7f233e6b8-metrics-certs\") pod \"network-metrics-daemon-f549x\" (UID: \"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8\") " pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:50.864647 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:50.864504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c4b166f-98c3-43d4-a8b6-6872182419f6-original-pull-secret\") pod \"global-pull-secret-syncer-g2nbg\" (UID: \"2c4b166f-98c3-43d4-a8b6-6872182419f6\") " pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:51.025914 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.025839 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47bt6\"" Jul 09 18:54:51.031294 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.031275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:51.035480 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.035455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkpmb\"" Jul 09 18:54:51.040121 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.040095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g2nbg" Jul 09 18:54:51.043249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.043228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f549x" Jul 09 18:54:51.170310 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.170287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hvxv"] Jul 09 18:54:51.172886 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:51.172859 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a434b2e_430b_4531_a9c5_15172d73e62a.slice/crio-a20bae98ee5c1ac566b7ba95ea9af507c8795b9a0b90e8e44c8aa1cac039e7cc WatchSource:0}: Error finding container a20bae98ee5c1ac566b7ba95ea9af507c8795b9a0b90e8e44c8aa1cac039e7cc: Status 404 returned error can't find the container with id a20bae98ee5c1ac566b7ba95ea9af507c8795b9a0b90e8e44c8aa1cac039e7cc Jul 09 18:54:51.370772 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.370661 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-87bbc467f-pzhnk"] Jul 09 18:54:51.372918 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.372894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.377367 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.377315 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 09 18:54:51.377483 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.377384 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g6l6l\"" Jul 09 18:54:51.378873 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.378745 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 18:54:51.378873 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.378768 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 09 18:54:51.378873 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.378772 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-731qt69ac1qsl\"" Jul 09 18:54:51.379085 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.378830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 09 18:54:51.395831 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.395807 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-87bbc467f-pzhnk"] Jul 09 18:54:51.406033 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.406008 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f549x"] Jul 09 18:54:51.407080 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.407066 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g2nbg"] Jul 09 18:54:51.407581 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.407556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hvxv" event={"ID":"0a434b2e-430b-4531-a9c5-15172d73e62a","Type":"ContainerStarted","Data":"a20bae98ee5c1ac566b7ba95ea9af507c8795b9a0b90e8e44c8aa1cac039e7cc"} Jul 09 18:54:51.409319 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.409295 2578 generic.go:356] "Generic (PLEG): container finished" podID="37952e3a-9fb9-4860-b3c6-d02429d75b58" containerID="ddd2c131b23b3f9edd890130a06b04f1af424f1213d6eac0f897fe4acfea9e5d" exitCode=0 Jul 09 18:54:51.409414 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:51.409321 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f1bca4_a35c_40c0_8591_b4a7f233e6b8.slice/crio-28eda7a0a86dac3cc19f7521a0774b348f5be032df439f8932adbb54bf7c847f WatchSource:0}: Error finding container 28eda7a0a86dac3cc19f7521a0774b348f5be032df439f8932adbb54bf7c847f: Status 404 returned error can't find the container with id 28eda7a0a86dac3cc19f7521a0774b348f5be032df439f8932adbb54bf7c847f Jul 09 18:54:51.409414 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.409383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h8zb" event={"ID":"37952e3a-9fb9-4860-b3c6-d02429d75b58","Type":"ContainerDied","Data":"ddd2c131b23b3f9edd890130a06b04f1af424f1213d6eac0f897fe4acfea9e5d"} Jul 09 18:54:51.410339 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:51.410235 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4b166f_98c3_43d4_a8b6_6872182419f6.slice/crio-b07fa50db0934fcf73901145134f8c093a231803151c6ecfc84b384a1ebd0045 WatchSource:0}: Error finding container b07fa50db0934fcf73901145134f8c093a231803151c6ecfc84b384a1ebd0045: Status 404 returned error can't find the container with id b07fa50db0934fcf73901145134f8c093a231803151c6ecfc84b384a1ebd0045 Jul 09 18:54:51.411814 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.411791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" event={"ID":"12911dea-c8cc-4a03-8ca7-7bfd2f269a1e","Type":"ContainerStarted","Data":"8f3dcca31098599c5e658fac1a5ef7b6bbe559bab22a83c5a085f54ea84869f4"} Jul 09 18:54:51.413323 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.413136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"8aa113a981efe897f5d6f866c5f67483556ee0a57425d487cff795013808beb0"} Jul 09 18:54:51.457272 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5zb\" (UniqueName: \"kubernetes.io/projected/b096a242-1d66-46e6-a7ae-48771b467891-kube-api-access-dq5zb\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457444 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457504 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-client-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457552 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-client-certs\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457616 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-metrics-server-audit-profiles\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b096a242-1d66-46e6-a7ae-48771b467891-audit-log\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.457808 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.457780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-tls\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.463291 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.463252 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-fzw6q" podStartSLOduration=3.980808201 podStartE2EDuration="5.463236406s" totalImagesPullingTime="1.482428205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:51.461423426 +0000 UTC m=+66.878137830" watchObservedRunningTime="2026-07-09 18:54:51.463236406 +0000 UTC m=+66.879950815" Jul 09 18:54:51.558635 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5zb\" (UniqueName: \"kubernetes.io/projected/b096a242-1d66-46e6-a7ae-48771b467891-kube-api-access-dq5zb\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.558800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.558800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-client-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.558800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-client-certs\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.558800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-metrics-server-audit-profiles\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.559026 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b096a242-1d66-46e6-a7ae-48771b467891-audit-log\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.559026 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.558873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-tls\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.560048 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.559971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b096a242-1d66-46e6-a7ae-48771b467891-audit-log\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.560048 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.559996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.560386 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.560363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b096a242-1d66-46e6-a7ae-48771b467891-metrics-server-audit-profiles\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.562509 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.561913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-client-ca-bundle\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.563011 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.562693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-tls\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.563011 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.562724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b096a242-1d66-46e6-a7ae-48771b467891-secret-metrics-server-client-certs\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.568606 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.568565 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5zb\" (UniqueName: \"kubernetes.io/projected/b096a242-1d66-46e6-a7ae-48771b467891-kube-api-access-dq5zb\") pod \"metrics-server-87bbc467f-pzhnk\" (UID: \"b096a242-1d66-46e6-a7ae-48771b467891\") " pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.576377 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.575197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:51.576377 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.575287 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:54:51.576697 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.576654 2578 patch_prober.go:28] interesting pod/console-6b8868bdb9-m8tqk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 09 18:54:51.576843 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.576780 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6b8868bdb9-m8tqk" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 09 18:54:51.682645 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.682614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:54:51.763106 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.763070 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-8n882"] Jul 09 18:54:51.766453 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.766427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.769843 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.769426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 18:54:51.769843 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.769667 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-67bw9\"" Jul 09 18:54:51.775302 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.775229 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-8n882"] Jul 09 18:54:51.851907 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.851699 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-87bbc467f-pzhnk"] Jul 09 18:54:51.854247 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:51.854220 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096a242_1d66_46e6_a7ae_48771b467891.slice/crio-bfe4674da5a35c6db8127503088b55e7603635cc55a9e8d76d7e8104fe3537f7 WatchSource:0}: Error finding container bfe4674da5a35c6db8127503088b55e7603635cc55a9e8d76d7e8104fe3537f7: Status 404 returned error can't find the container with id bfe4674da5a35c6db8127503088b55e7603635cc55a9e8d76d7e8104fe3537f7 Jul 09 18:54:51.861444 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.861394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e28520c9-0718-4e99-a006-98a018a7e90b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.861549 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.861458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9wj\" (UniqueName: \"kubernetes.io/projected/e28520c9-0718-4e99-a006-98a018a7e90b-kube-api-access-mp9wj\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.962375 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.962284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e28520c9-0718-4e99-a006-98a018a7e90b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.962375 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.962357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9wj\" (UniqueName: \"kubernetes.io/projected/e28520c9-0718-4e99-a006-98a018a7e90b-kube-api-access-mp9wj\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.965545 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.965496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e28520c9-0718-4e99-a006-98a018a7e90b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:51.973234 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:51.973187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9wj\" (UniqueName: \"kubernetes.io/projected/e28520c9-0718-4e99-a006-98a018a7e90b-kube-api-access-mp9wj\") pod \"monitoring-plugin-5f68df7f58-8n882\" (UID: \"e28520c9-0718-4e99-a006-98a018a7e90b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:52.079029 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.078996 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:52.258293 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.258207 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-8n882"] Jul 09 18:54:52.303177 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.303087 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:54:52.418160 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.418122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" event={"ID":"b096a242-1d66-46e6-a7ae-48771b467891","Type":"ContainerStarted","Data":"bfe4674da5a35c6db8127503088b55e7603635cc55a9e8d76d7e8104fe3537f7"} Jul 09 18:54:52.418160 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:52.418156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28520c9_0718_4e99_a006_98a018a7e90b.slice/crio-f1dbf3dcedf11cc05e1dfc5022b7be927dc6f816de230256c67a156bc894af42 WatchSource:0}: Error finding container f1dbf3dcedf11cc05e1dfc5022b7be927dc6f816de230256c67a156bc894af42: Status 404 returned error can't find the container with id f1dbf3dcedf11cc05e1dfc5022b7be927dc6f816de230256c67a156bc894af42 Jul 09 18:54:52.421254 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.420931 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h8zb" event={"ID":"37952e3a-9fb9-4860-b3c6-d02429d75b58","Type":"ContainerStarted","Data":"d078eb7bbcaccc69af1cb22e324dc03d6dbed7ba8c3ae7b262c3a86c868913e1"} Jul 09 18:54:52.421254 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.420979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h8zb" event={"ID":"37952e3a-9fb9-4860-b3c6-d02429d75b58","Type":"ContainerStarted","Data":"87fb2ece4dfbc0e66d6eb7467e45b309fcf071d3889171ba33633d51f80d0de8"} Jul 09 18:54:52.422668 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.422601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f549x" event={"ID":"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8","Type":"ContainerStarted","Data":"28eda7a0a86dac3cc19f7521a0774b348f5be032df439f8932adbb54bf7c847f"} Jul 09 18:54:52.424201 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.424013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g2nbg" event={"ID":"2c4b166f-98c3-43d4-a8b6-6872182419f6","Type":"ContainerStarted","Data":"b07fa50db0934fcf73901145134f8c093a231803151c6ecfc84b384a1ebd0045"} Jul 09 18:54:52.449960 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:52.449910 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8h8zb" podStartSLOduration=5.298019583 podStartE2EDuration="6.449890478s" totalImagesPullingTime="1.080161699s" totalInitContainerRuntime="71.709196ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:52.448824761 +0000 UTC m=+67.865539184" watchObservedRunningTime="2026-07-09 18:54:52.449890478 +0000 UTC m=+67.866604889" Jul 09 18:54:53.225590 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.225554 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:54:53.250978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.250942 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:54:53.251144 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.251118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.254648 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.254450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 18:54:53.254880 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.254863 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 18:54:53.255097 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.255078 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 18:54:53.255314 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.255296 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w278k\"" Jul 09 18:54:53.255540 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.255521 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 18:54:53.256049 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.256023 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 18:54:53.256262 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.256247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 18:54:53.256389 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.256360 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 18:54:53.256453 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.256445 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 18:54:53.256562 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.256252 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 18:54:53.258366 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.258278 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 18:54:53.258519 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.258488 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 18:54:53.260348 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.260326 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 18:54:53.260571 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.260541 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8nsado73gqf6t\"" Jul 09 18:54:53.262354 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.262324 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 18:54:53.268302 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.265160 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 18:54:53.374612 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374812 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374812 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374812 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.374978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.374991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28tg\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375255 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.375475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.375353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.429451 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.429409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" event={"ID":"e28520c9-0718-4e99-a006-98a018a7e90b","Type":"ContainerStarted","Data":"f1dbf3dcedf11cc05e1dfc5022b7be927dc6f816de230256c67a156bc894af42"} Jul 09 18:54:53.476325 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476325 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476530 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28tg\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.476920 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.477249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.477249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.477249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.477249 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.476997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.477758 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.477663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.478362 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.478332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.480768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.481048 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.481349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.481810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.482316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.482672 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.483706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.484237 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.484306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.484760 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.486847 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.485993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.487418 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.487276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.487897 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.487852 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.488134 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.488094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.488315 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.488294 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.488409 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.488388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.502099 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.502045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28tg\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg\") pod \"prometheus-k8s-0\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:53.567796 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:53.567749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:54:55.379766 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:55.379710 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ctn98" Jul 09 18:54:56.743633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:56.743590 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:54:56.979976 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:54:56.979935 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6fe3b7_b415_475c_935d_1d209356cba9.slice/crio-a49a9c7b513153d6c00659f2cff62badeedb3f890a8709c98cdf2016b5a5c7d8 WatchSource:0}: Error finding container a49a9c7b513153d6c00659f2cff62badeedb3f890a8709c98cdf2016b5a5c7d8: Status 404 returned error can't find the container with id a49a9c7b513153d6c00659f2cff62badeedb3f890a8709c98cdf2016b5a5c7d8 Jul 09 18:54:57.451934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.451892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" event={"ID":"b096a242-1d66-46e6-a7ae-48771b467891","Type":"ContainerStarted","Data":"1453a017b6f508b20df28887288eb23ae6b75c6c66a3a0e9c05885018b32286c"} Jul 09 18:54:57.453880 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.453847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hvxv" event={"ID":"0a434b2e-430b-4531-a9c5-15172d73e62a","Type":"ContainerStarted","Data":"3d9f3f049f1a0c29708e2fd38a053ead1b22c209cdb61b889a52065f8241a9da"} Jul 09 18:54:57.454032 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.453920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:54:57.455273 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.455251 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" event={"ID":"e28520c9-0718-4e99-a006-98a018a7e90b","Type":"ContainerStarted","Data":"b67386302b977ebb034323f6e7b303b0fd4c2bdec29887fb545287ae5f90db2e"} Jul 09 18:54:57.455411 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.455391 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:57.456531 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.456509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"a49a9c7b513153d6c00659f2cff62badeedb3f890a8709c98cdf2016b5a5c7d8"} Jul 09 18:54:57.458358 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.458331 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f549x" event={"ID":"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8","Type":"ContainerStarted","Data":"2606e5baa9f21d5eca35c623d8dd4c5f1dde02359836d90e07d6c576355265fd"} Jul 09 18:54:57.458473 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.458380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f549x" event={"ID":"f2f1bca4-a35c-40c0-8591-b4a7f233e6b8","Type":"ContainerStarted","Data":"14dfbe72ff81e6554c19aab327140f493994d21dc9dc16b3be805cac2bb924fe"} Jul 09 18:54:57.460620 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.460592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"012a1dbb6f009209ce59043911edc27bfb947bd7f1497652beebb81548dba23e"} Jul 09 18:54:57.460724 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.460660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"e852a18e503bd945c82a1685689984a0b02361f68ca5d4d05574ed3d410e84a7"} Jul 09 18:54:57.460724 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.460696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"72177987c3fcb215f2c3f4a9e6f05213410a99dba7ebc3b40fe6aa8990a67cc9"} Jul 09 18:54:57.460995 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.460974 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" Jul 09 18:54:57.462137 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.462118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g2nbg" event={"ID":"2c4b166f-98c3-43d4-a8b6-6872182419f6","Type":"ContainerStarted","Data":"d920a40b014176d1ef80e11045cb1e4e275901d963c62bad383441884f9ab6c3"} Jul 09 18:54:57.474671 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.474620 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" podStartSLOduration=1.739213466 podStartE2EDuration="6.4746048s" totalImagesPullingTime="4.735391334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.473903521 +0000 UTC m=+72.890617930" watchObservedRunningTime="2026-07-09 18:54:57.4746048 +0000 UTC m=+72.891319210" Jul 09 18:54:57.498120 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.498006 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7hvxv" podStartSLOduration=67.071100579 podStartE2EDuration="1m12.497989017s" totalImagesPullingTime="5.426888438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.489675699 +0000 UTC m=+72.906390110" watchObservedRunningTime="2026-07-09 18:54:57.497989017 +0000 UTC m=+72.914703405" Jul 09 18:54:57.510410 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.510346 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g2nbg" podStartSLOduration=66.923591403 podStartE2EDuration="1m12.510324002s" totalImagesPullingTime="5.586732599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.510018996 +0000 UTC m=+72.926733419" watchObservedRunningTime="2026-07-09 18:54:57.510324002 +0000 UTC m=+72.927038423" Jul 09 18:54:57.528418 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.528359 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f549x" podStartSLOduration=71.465907281 podStartE2EDuration="1m12.528342696s" totalImagesPullingTime="1.062435415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.527958344 +0000 UTC m=+72.944672757" watchObservedRunningTime="2026-07-09 18:54:57.528342696 +0000 UTC m=+72.945057183" Jul 09 18:54:57.546475 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:57.546421 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-8n882" podStartSLOduration=1.91952908 podStartE2EDuration="6.546405494s" totalImagesPullingTime="4.626876414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.546020009 +0000 UTC m=+72.962734420" watchObservedRunningTime="2026-07-09 18:54:57.546405494 +0000 UTC m=+72.963119904" Jul 09 18:54:58.468626 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:58.468592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} Jul 09 18:54:58.471268 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:58.471196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"93d8914db4e33bb6eb74f250f115b03f08908130178d873429593adf4a9f9e27"} Jul 09 18:54:59.477284 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:59.477243 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" exitCode=0 Jul 09 18:54:59.477781 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:59.477318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} Jul 09 18:54:59.480881 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:59.480854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"dae943c5bcf6e8236aaa8d8533728e00643a8374251af959eba5e44f41d33a95"} Jul 09 18:54:59.481000 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:59.480927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" event={"ID":"fa81b2e5-bab7-468b-b239-414689dad316","Type":"ContainerStarted","Data":"65efd77ffa8bbd10d5f579aad583396cdeb07324277ced6ce74a89be2daabf3a"} Jul 09 18:54:59.512713 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:54:59.512653 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" podStartSLOduration=2.293162831 podStartE2EDuration="9.512634079s" totalImagesPullingTime="7.219471248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:59.511835455 +0000 UTC m=+74.928549865" watchObservedRunningTime="2026-07-09 18:54:59.512634079 +0000 UTC m=+74.929348519" Jul 09 18:55:00.485561 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:00.485528 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:55:02.501860 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.501826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} Jul 09 18:55:02.502238 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.501913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} Jul 09 18:55:02.502238 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.501939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} Jul 09 18:55:02.502238 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.501963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} Jul 09 18:55:02.502238 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.501984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} Jul 09 18:55:02.502238 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.502007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerStarted","Data":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} Jul 09 18:55:02.540969 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:02.540899 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.935873964 podStartE2EDuration="9.540878098s" totalImagesPullingTime="4.582949877s" totalInitContainerRuntime="22.054257ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:55:02.538824375 +0000 UTC m=+77.955538776" watchObservedRunningTime="2026-07-09 18:55:02.540878098 +0000 UTC m=+77.957592508" Jul 09 18:55:03.568045 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:03.567990 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:55:06.496259 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:06.496229 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-799b685df8-sxvgk" Jul 09 18:55:11.683274 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:11.683236 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:55:11.683718 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:11.683298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:55:15.084812 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:15.084786 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9r5g_16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb/serve-healthcheck-canary/0.log" Jul 09 18:55:17.447934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:17.447892 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b8868bdb9-m8tqk" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerName="console" containerID="cri-o://11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc" gracePeriod=15 Jul 09 18:55:18.233900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.233863 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:55:18.392523 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392493 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392533 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vp5f\" (UniqueName: \"kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392569 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392622 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.392703 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.392696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle\") pod \"9d8f78ce-0498-4786-8af3-0d34a883eff5\" (UID: \"9d8f78ce-0498-4786-8af3-0d34a883eff5\") " Jul 09 18:55:18.393046 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.393018 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:18.393098 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.393069 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:18.393195 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.393172 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config" (OuterVolumeSpecName: "console-config") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:18.393260 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.393198 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:55:18.395029 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.395000 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:55:18.395139 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.395065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f" (OuterVolumeSpecName: "kube-api-access-4vp5f") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "kube-api-access-4vp5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:55:18.395139 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.395086 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d8f78ce-0498-4786-8af3-0d34a883eff5" (UID: "9d8f78ce-0498-4786-8af3-0d34a883eff5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:55:18.493633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493596 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-trusted-ca-bundle\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.493633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493626 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-serving-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.493633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493639 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vp5f\" (UniqueName: \"kubernetes.io/projected/9d8f78ce-0498-4786-8af3-0d34a883eff5-kube-api-access-4vp5f\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.494092 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493653 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-oauth-serving-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.494092 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493664 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-oauth-config\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.494092 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493675 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-service-ca\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.494092 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.493687 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8f78ce-0498-4786-8af3-0d34a883eff5-console-config\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:55:18.562382 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.562348 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerID="11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc" exitCode=0 Jul 09 18:55:18.562565 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.562394 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8868bdb9-m8tqk" Jul 09 18:55:18.562565 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.562411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8868bdb9-m8tqk" event={"ID":"9d8f78ce-0498-4786-8af3-0d34a883eff5","Type":"ContainerDied","Data":"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc"} Jul 09 18:55:18.562565 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.562461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8868bdb9-m8tqk" event={"ID":"9d8f78ce-0498-4786-8af3-0d34a883eff5","Type":"ContainerDied","Data":"5e5941d1dc680f25da9ecad553f55ab0548671ca2a95d7233c57958b69159ec9"} Jul 09 18:55:18.562565 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.562486 2578 scope.go:117] "RemoveContainer" containerID="11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc" Jul 09 18:55:18.572651 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.572629 2578 scope.go:117] "RemoveContainer" containerID="11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc" Jul 09 18:55:18.573140 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:55:18.573117 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc\": container with ID starting with 11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc not found: ID does not exist" containerID="11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc" Jul 09 18:55:18.573227 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.573147 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc"} err="failed to get container status \"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc\": rpc error: code = NotFound desc = could not find container \"11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc\": container with ID starting with 11341f202da03b5057cd0cf3938a4f2248e8905bbe3542b20171748a247433cc not found: ID does not exist" Jul 09 18:55:18.585959 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.585936 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:55:18.590425 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:18.590405 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b8868bdb9-m8tqk"] Jul 09 18:55:19.105188 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:19.105156 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" path="/var/lib/kubelet/pods/9d8f78ce-0498-4786-8af3-0d34a883eff5/volumes" Jul 09 18:55:28.474812 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:28.474351 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7hvxv" Jul 09 18:55:31.688689 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:31.688656 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:55:31.692622 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:31.692600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-87bbc467f-pzhnk" Jul 09 18:55:53.568036 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:53.568003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:55:53.586701 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:53.586676 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:55:53.703797 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:55:53.703772 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:11.668957 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.668921 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669207 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="prometheus" containerID="cri-o://48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" gracePeriod=600 Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669224 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="thanos-sidecar" containerID="cri-o://5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" gracePeriod=600 Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669234 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy" containerID="cri-o://b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" gracePeriod=600 Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669294 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="config-reloader" containerID="cri-o://1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" gracePeriod=600 Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669348 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" gracePeriod=600 Jul 09 18:56:11.669464 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.669296 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-web" containerID="cri-o://72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" gracePeriod=600 Jul 09 18:56:11.899697 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.899672 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:11.997127 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997036 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997127 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997127 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997105 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997162 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c28tg\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997292 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997341 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997400 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997386 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997774 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997421 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997774 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.997774 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:11.997934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:11.997934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:11.998037 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998037 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.997997 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998037 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998027 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998184 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998067 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998184 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998094 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998184 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998184 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998154 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998361 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998180 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998361 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998215 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs\") pod \"6c6fe3b7-b415-475c-935d-1d209356cba9\" (UID: \"6c6fe3b7-b415-475c-935d-1d209356cba9\") " Jul 09 18:56:11.998361 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998284 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:11.998839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998648 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:11.998839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998674 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:11.998839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998691 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:11.998839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998706 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:11.999098 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.998959 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:11.999829 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.999782 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:12.000169 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:11.999958 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out" (OuterVolumeSpecName: "config-out") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:56:12.000169 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.000033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 18:56:12.000619 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.000524 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:56:12.000755 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.000695 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg" (OuterVolumeSpecName: "kube-api-access-c28tg") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "kube-api-access-c28tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:56:12.001211 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.001181 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.001298 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.001249 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002278 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002251 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002487 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002457 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002599 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002536 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002599 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002586 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config" (OuterVolumeSpecName: "config") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002699 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002681 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 18:56:12.002772 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002699 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.002823 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.002787 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.010805 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.010762 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config" (OuterVolumeSpecName: "web-config") pod "6c6fe3b7-b415-475c-935d-1d209356cba9" (UID: "6c6fe3b7-b415-475c-935d-1d209356cba9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:56:12.099396 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099351 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099396 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099393 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099396 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099405 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099396 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099415 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-db\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099426 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-configmap-metrics-client-ca\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099434 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-tls-assets\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099444 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-grpc-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099459 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099469 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-metrics-client-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099477 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6c6fe3b7-b415-475c-935d-1d209356cba9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099485 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c6fe3b7-b415-475c-935d-1d209356cba9-config-out\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099494 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-web-config\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099501 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28tg\" (UniqueName: \"kubernetes.io/projected/6c6fe3b7-b415-475c-935d-1d209356cba9-kube-api-access-c28tg\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099510 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-secret-kube-rbac-proxy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099518 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-config\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.099674 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.099526 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6c6fe3b7-b415-475c-935d-1d209356cba9-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 18:56:12.754759 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754705 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" exitCode=0 Jul 09 18:56:12.754759 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754751 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" exitCode=0 Jul 09 18:56:12.754759 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754763 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" exitCode=0 Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754772 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" exitCode=0 Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754811 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754780 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" exitCode=0 Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754866 2578 generic.go:356] "Generic (PLEG): container finished" podID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" exitCode=0 Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754872 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.755000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6c6fe3b7-b415-475c-935d-1d209356cba9","Type":"ContainerDied","Data":"a49a9c7b513153d6c00659f2cff62badeedb3f890a8709c98cdf2016b5a5c7d8"} Jul 09 18:56:12.755212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.754946 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.763201 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.763173 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.769594 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.769579 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.775845 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.775827 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.780457 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.780439 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:12.783248 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.783232 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.784850 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.784831 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:12.789501 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.789487 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.796186 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.796169 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.802485 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.802470 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.802781 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.802752 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.803033 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.802792 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.803033 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.802813 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.803119 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.803055 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.803119 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803073 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.803119 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803086 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.803302 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.803287 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.803336 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803305 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.803336 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803318 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.803552 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.803534 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.803602 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803555 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.803602 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803569 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.803785 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.803770 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.803833 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803788 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.803833 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.803799 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.803993 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.803976 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.804052 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804000 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.804052 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804022 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.804265 ip-10-0-131-3 kubenswrapper[2578]: E0709 18:56:12.804245 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.804305 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804270 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.804305 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804285 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.804492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804472 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.804492 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804492 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.804743 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804710 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.804793 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804747 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.804950 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804931 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.804992 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.804950 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.805157 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805142 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.805198 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805166 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.805370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805348 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.805455 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805370 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.805603 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805588 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.805646 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805603 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.805916 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805896 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.805975 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.805918 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.806174 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806156 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.806240 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806177 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.806386 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806369 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.806435 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806387 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.806620 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806602 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.806620 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806620 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.806839 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806820 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.806889 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.806840 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.807016 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807000 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.807057 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807017 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.807244 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807227 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.807244 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807243 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.807466 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807449 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.807512 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807467 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.807651 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807637 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.807692 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807651 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.807925 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807903 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.807970 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.807927 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.808121 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808104 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.808172 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808122 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.808370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808349 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.808370 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808369 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.808602 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808580 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.808602 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808599 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.808948 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808912 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.808948 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.808940 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.809212 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809190 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.809275 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809214 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.809472 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809447 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.809472 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809471 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.809795 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.809795 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.809791 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.810046 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810022 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.810129 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810047 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.810311 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810289 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.810359 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810314 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.810393 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810376 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:12.810550 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810532 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.810623 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810553 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.810666 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810641 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="config-reloader" Jul 09 18:56:12.810666 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810655 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="config-reloader" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810676 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810684 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810697 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="thanos-sidecar" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810705 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="thanos-sidecar" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810716 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-web" Jul 09 18:56:12.810746 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810744 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-web" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810757 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-thanos" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810765 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-thanos" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810777 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="init-config-reloader" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810783 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="init-config-reloader" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810794 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerName="console" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810803 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerName="console" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810805 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810830 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810814 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="prometheus" Jul 09 18:56:12.810934 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810878 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="prometheus" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.810971 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="config-reloader" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811009 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="prometheus" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811016 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-thanos" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811022 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="thanos-sidecar" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811027 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy-web" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811032 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" containerName="kube-rbac-proxy" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811038 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8f78ce-0498-4786-8af3-0d34a883eff5" containerName="console" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811066 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.811303 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811087 2578 scope.go:117] "RemoveContainer" containerID="343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a" Jul 09 18:56:12.811633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811316 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a"} err="failed to get container status \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": rpc error: code = NotFound desc = could not find container \"343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a\": container with ID starting with 343aa34eee1e8d6a96a68ef81425b1cefbdae0840372698f06f8715fad37df6a not found: ID does not exist" Jul 09 18:56:12.811633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811333 2578 scope.go:117] "RemoveContainer" containerID="b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476" Jul 09 18:56:12.811633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811595 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476"} err="failed to get container status \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": rpc error: code = NotFound desc = could not find container \"b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476\": container with ID starting with b643c9c0ed702eb0457a1359a1281923ade7b49b516f545571b6b7691f4b7476 not found: ID does not exist" Jul 09 18:56:12.811633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811608 2578 scope.go:117] "RemoveContainer" containerID="72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85" Jul 09 18:56:12.811878 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811858 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85"} err="failed to get container status \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": rpc error: code = NotFound desc = could not find container \"72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85\": container with ID starting with 72b7dab2ea618b591ac4feeb972d77599d77eb84bb4773f34ab8308c207fbd85 not found: ID does not exist" Jul 09 18:56:12.811925 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.811878 2578 scope.go:117] "RemoveContainer" containerID="5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c" Jul 09 18:56:12.812104 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812087 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c"} err="failed to get container status \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": rpc error: code = NotFound desc = could not find container \"5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c\": container with ID starting with 5bb401a6bb6e554f36dddf5153565dc49989dacb1b5838c24765a46f56521b0c not found: ID does not exist" Jul 09 18:56:12.812150 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812105 2578 scope.go:117] "RemoveContainer" containerID="1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8" Jul 09 18:56:12.812331 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812307 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8"} err="failed to get container status \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": rpc error: code = NotFound desc = could not find container \"1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8\": container with ID starting with 1926ffb8798b2d470fb3934a86259e312cc8fe42080061ae03f4850737a7c9a8 not found: ID does not exist" Jul 09 18:56:12.812377 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812332 2578 scope.go:117] "RemoveContainer" containerID="48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b" Jul 09 18:56:12.812557 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812540 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b"} err="failed to get container status \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": rpc error: code = NotFound desc = could not find container \"48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b\": container with ID starting with 48e8bf78de602d503d43ea3326bf8c5ba0ed94b75c041296f658442ae500925b not found: ID does not exist" Jul 09 18:56:12.812604 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812557 2578 scope.go:117] "RemoveContainer" containerID="9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e" Jul 09 18:56:12.812799 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.812784 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e"} err="failed to get container status \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": rpc error: code = NotFound desc = could not find container \"9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e\": container with ID starting with 9624bf3c638f17031283a504163ffb38039b4c95f1779d675761e1940aeb521e not found: ID does not exist" Jul 09 18:56:12.816201 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.816187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.818975 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.818958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 18:56:12.819407 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819387 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 18:56:12.819557 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819541 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 18:56:12.819642 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819627 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 18:56:12.819780 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819767 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 18:56:12.819850 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819779 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 18:56:12.820014 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.819998 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 18:56:12.820072 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.820031 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 18:56:12.820072 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.820054 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 18:56:12.820072 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.820060 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 18:56:12.820695 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.820675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 18:56:12.820898 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.820883 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 18:56:12.821457 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.821441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8nsado73gqf6t\"" Jul 09 18:56:12.821611 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.821594 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w278k\"" Jul 09 18:56:12.824164 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.824145 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 18:56:12.825338 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.825321 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 18:56:12.832600 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.832580 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:12.906394 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906487 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906487 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906487 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906487 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906643 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.906900 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgzg\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-kube-api-access-9qgzg\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:12.907109 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:12.906904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007825 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007825 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007988 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007988 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007988 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007988 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.007988 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008200 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.007987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008200 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008665 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008815 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008884 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008884 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008999 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008999 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008999 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.008999 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.008977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgzg\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-kube-api-access-9qgzg\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009191 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009482 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.009901 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.010020 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.009995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.010665 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.010637 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.010767 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.010709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.010980 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.010958 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.010980 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.010975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.011828 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.011482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.012419 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.012392 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.012504 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.012466 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.013251 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.013226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.013425 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.013399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.013549 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.013524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.013707 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.013688 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.014166 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.014132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.014437 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.014422 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.019477 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.019456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgzg\" (UniqueName: \"kubernetes.io/projected/89ef756e-496a-4a1a-8fb3-8d4c73cdae6b-kube-api-access-9qgzg\") pod \"prometheus-k8s-0\" (UID: \"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.104565 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.104532 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6fe3b7-b415-475c-935d-1d209356cba9" path="/var/lib/kubelet/pods/6c6fe3b7-b415-475c-935d-1d209356cba9/volumes" Jul 09 18:56:13.126019 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.125995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:56:13.265569 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.265487 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 18:56:13.269400 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:56:13.269362 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ef756e_496a_4a1a_8fb3_8d4c73cdae6b.slice/crio-7e6eaf3dbd6edd32437d21ff71c283cd1e09fced83975f3701e971d38ba19376 WatchSource:0}: Error finding container 7e6eaf3dbd6edd32437d21ff71c283cd1e09fced83975f3701e971d38ba19376: Status 404 returned error can't find the container with id 7e6eaf3dbd6edd32437d21ff71c283cd1e09fced83975f3701e971d38ba19376 Jul 09 18:56:13.760162 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.760128 2578 generic.go:356] "Generic (PLEG): container finished" podID="89ef756e-496a-4a1a-8fb3-8d4c73cdae6b" containerID="eda256358d0175c1c18aaa2327b979934fa93894a8d581571ceb1696aded3601" exitCode=0 Jul 09 18:56:13.760597 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.760221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerDied","Data":"eda256358d0175c1c18aaa2327b979934fa93894a8d581571ceb1696aded3601"} Jul 09 18:56:13.760597 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:13.760312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"7e6eaf3dbd6edd32437d21ff71c283cd1e09fced83975f3701e971d38ba19376"} Jul 09 18:56:14.770837 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.770801 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"fe318a4ac32f6d72ce91e2b6ce1ff3d42193f1a711fdac6a5799ba22d1ec3f89"} Jul 09 18:56:14.771217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.770897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"9b7b45fb0e4219ff246611c9258e191a8f713e72ac91f9f02362172ee8df252d"} Jul 09 18:56:14.771217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.770936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"5f6cf22eaf553ca6eb399b562d99bc1ce29b47d79035e07ae480edc0de3fc184"} Jul 09 18:56:14.771217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.770965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"cc2c1755c2c1f86e5a660cdc77f322fdca7a9dcc7c2f8c09c01c86c5b9f977ac"} Jul 09 18:56:14.771217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.770989 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"88c3a6f74331d1c21318304cc6b9e80fc3773c97a63c3d0a7c25a84d53570754"} Jul 09 18:56:14.771217 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.771023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ef756e-496a-4a1a-8fb3-8d4c73cdae6b","Type":"ContainerStarted","Data":"cd83cd6857d20b15cab654b72c149ca97710a30dc7dd0083820711131fdaff09"} Jul 09 18:56:14.816273 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:14.816215 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.794995823 podStartE2EDuration="2.816194719s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.198896ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:56:14.813951669 +0000 UTC m=+150.230666091" watchObservedRunningTime="2026-07-09 18:56:14.816194719 +0000 UTC m=+150.232909130" Jul 09 18:56:18.126471 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:56:18.126424 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:57:13.126362 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:13.126325 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:57:13.141482 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:13.141457 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:57:13.994796 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:13.994710 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 18:57:51.107035 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.107006 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h56hh"] Jul 09 18:57:51.115521 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.115493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.118435 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.118416 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 18:57:51.118695 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.118661 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 18:57:51.118695 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.118665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-66f5v\"" Jul 09 18:57:51.122037 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.122012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h56hh"] Jul 09 18:57:51.154752 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.154708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwnw\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-kube-api-access-qqwnw\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.154886 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.154824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-bound-sa-token\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.256058 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.256030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwnw\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-kube-api-access-qqwnw\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.256201 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.256084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-bound-sa-token\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.265066 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.265044 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-bound-sa-token\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.265216 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.265196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwnw\" (UniqueName: \"kubernetes.io/projected/c6dcb84b-d7d6-4196-aafa-1c1726d46234-kube-api-access-qqwnw\") pod \"cert-manager-55b4d954d5-h56hh\" (UID: \"c6dcb84b-d7d6-4196-aafa-1c1726d46234\") " pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.437167 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.437122 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h56hh" Jul 09 18:57:51.560036 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:51.560014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h56hh"] Jul 09 18:57:51.562601 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:57:51.562560 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6dcb84b_d7d6_4196_aafa_1c1726d46234.slice/crio-37534801e54e1799222150a581e941e27dbb9d1fcb7d119205794fa6375be40d WatchSource:0}: Error finding container 37534801e54e1799222150a581e941e27dbb9d1fcb7d119205794fa6375be40d: Status 404 returned error can't find the container with id 37534801e54e1799222150a581e941e27dbb9d1fcb7d119205794fa6375be40d Jul 09 18:57:52.109413 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:52.109379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h56hh" event={"ID":"c6dcb84b-d7d6-4196-aafa-1c1726d46234","Type":"ContainerStarted","Data":"37534801e54e1799222150a581e941e27dbb9d1fcb7d119205794fa6375be40d"} Jul 09 18:57:54.138809 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.138778 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx"] Jul 09 18:57:54.142300 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.142276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.146440 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.146380 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jrbnv\"" Jul 09 18:57:54.146440 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.146385 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:57:54.146440 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.146422 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 18:57:54.151500 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.151451 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx"] Jul 09 18:57:54.178640 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.178612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r62l\" (UniqueName: \"kubernetes.io/projected/cbce2148-883b-44e4-8ab2-63a9624bcc1f-kube-api-access-2r62l\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.178796 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.178666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce2148-883b-44e4-8ab2-63a9624bcc1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.280068 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.280032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce2148-883b-44e4-8ab2-63a9624bcc1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.280234 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.280126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r62l\" (UniqueName: \"kubernetes.io/projected/cbce2148-883b-44e4-8ab2-63a9624bcc1f-kube-api-access-2r62l\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.280457 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.280434 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbce2148-883b-44e4-8ab2-63a9624bcc1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.289722 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.289693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r62l\" (UniqueName: \"kubernetes.io/projected/cbce2148-883b-44e4-8ab2-63a9624bcc1f-kube-api-access-2r62l\") pod \"openshift-lws-operator-fd8ccff4c-qlfqx\" (UID: \"cbce2148-883b-44e4-8ab2-63a9624bcc1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.454263 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.454172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" Jul 09 18:57:54.718505 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:54.718477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx"] Jul 09 18:57:54.721797 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:57:54.721766 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbce2148_883b_44e4_8ab2_63a9624bcc1f.slice/crio-5911fdbd6c789cb257db3d27754a05e7d6b83d0afdbdfcbddb78d8d779c65d2c WatchSource:0}: Error finding container 5911fdbd6c789cb257db3d27754a05e7d6b83d0afdbdfcbddb78d8d779c65d2c: Status 404 returned error can't find the container with id 5911fdbd6c789cb257db3d27754a05e7d6b83d0afdbdfcbddb78d8d779c65d2c Jul 09 18:57:55.122381 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:55.122289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h56hh" event={"ID":"c6dcb84b-d7d6-4196-aafa-1c1726d46234","Type":"ContainerStarted","Data":"73e22b2f975b64e75a38140da57ccff936c192d62092688ae7cff3cd507e2ad4"} Jul 09 18:57:55.125955 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:55.125920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" event={"ID":"cbce2148-883b-44e4-8ab2-63a9624bcc1f","Type":"ContainerStarted","Data":"5911fdbd6c789cb257db3d27754a05e7d6b83d0afdbdfcbddb78d8d779c65d2c"} Jul 09 18:57:55.139846 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:55.139796 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-h56hh" podStartSLOduration=1.047994548 podStartE2EDuration="4.139775838s" totalImagesPullingTime="3.09178129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:55.139556016 +0000 UTC m=+250.556270426" watchObservedRunningTime="2026-07-09 18:57:55.139775838 +0000 UTC m=+250.556490252" Jul 09 18:57:58.139092 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:58.139057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" event={"ID":"cbce2148-883b-44e4-8ab2-63a9624bcc1f","Type":"ContainerStarted","Data":"38fd45713c2055f6c0920a0248d41714009fdea77ab63ecf5ade87945f010364"} Jul 09 18:57:58.159659 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:57:58.159606 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qlfqx" podStartSLOduration=1.5967812609999998 podStartE2EDuration="4.159588745s" totalImagesPullingTime="2.562807484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:57:58.158431021 +0000 UTC m=+253.575145456" watchObservedRunningTime="2026-07-09 18:57:58.159588745 +0000 UTC m=+253.576303151" Jul 09 18:58:13.523215 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.523134 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4"] Jul 09 18:58:13.526614 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.526594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.529403 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.529382 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 18:58:13.529713 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.529691 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 18:58:13.529804 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.529759 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xxtqd\"" Jul 09 18:58:13.529856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.529816 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 18:58:13.529856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.529760 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 18:58:13.542381 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.542357 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4"] Jul 09 18:58:13.620300 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.620269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.620300 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.620307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.620523 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.620337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.620523 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.620391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.620523 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.620409 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkz5\" (UniqueName: \"kubernetes.io/projected/f66acc91-f60d-4884-91c5-a8f485d9a1f7-kube-api-access-jpkz5\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721428 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721428 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721681 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721681 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721681 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkz5\" (UniqueName: \"kubernetes.io/projected/f66acc91-f60d-4884-91c5-a8f485d9a1f7-kube-api-access-jpkz5\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.721967 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.721945 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.723588 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.723559 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66acc91-f60d-4884-91c5-a8f485d9a1f7-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.723911 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.723891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.724003 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.723963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f66acc91-f60d-4884-91c5-a8f485d9a1f7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.731638 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.731615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkz5\" (UniqueName: \"kubernetes.io/projected/f66acc91-f60d-4884-91c5-a8f485d9a1f7-kube-api-access-jpkz5\") pod \"opendatahub-operator-controller-manager-5cd98cb899-gm7r4\" (UID: \"f66acc91-f60d-4884-91c5-a8f485d9a1f7\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.836771 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.836662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:13.990599 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:13.990564 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4"] Jul 09 18:58:13.994600 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:58:13.994571 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66acc91_f60d_4884_91c5_a8f485d9a1f7.slice/crio-d792d9271a462b75b4d0d879f32e642b6acca9ad26a31d523ce1d73b8fa96452 WatchSource:0}: Error finding container d792d9271a462b75b4d0d879f32e642b6acca9ad26a31d523ce1d73b8fa96452: Status 404 returned error can't find the container with id d792d9271a462b75b4d0d879f32e642b6acca9ad26a31d523ce1d73b8fa96452 Jul 09 18:58:14.194875 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:14.194830 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" event={"ID":"f66acc91-f60d-4884-91c5-a8f485d9a1f7","Type":"ContainerStarted","Data":"d792d9271a462b75b4d0d879f32e642b6acca9ad26a31d523ce1d73b8fa96452"} Jul 09 18:58:17.208954 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:17.208910 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" event={"ID":"f66acc91-f60d-4884-91c5-a8f485d9a1f7","Type":"ContainerStarted","Data":"ede1be25de4d972fe7896d2dd7e54fe79b2c80d41eab633cd3bf9d5174fb6783"} Jul 09 18:58:18.213972 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:18.213936 2578 generic.go:356] "Generic (PLEG): container finished" podID="f66acc91-f60d-4884-91c5-a8f485d9a1f7" containerID="ede1be25de4d972fe7896d2dd7e54fe79b2c80d41eab633cd3bf9d5174fb6783" exitCode=0 Jul 09 18:58:18.214347 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:18.214020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" event={"ID":"f66acc91-f60d-4884-91c5-a8f485d9a1f7","Type":"ContainerDied","Data":"ede1be25de4d972fe7896d2dd7e54fe79b2c80d41eab633cd3bf9d5174fb6783"} Jul 09 18:58:19.219896 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:19.219860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" event={"ID":"f66acc91-f60d-4884-91c5-a8f485d9a1f7","Type":"ContainerStarted","Data":"4b67a96f6e50dc1889559cc0552edb2fcf567b1897773b308b8c5c2d8e66d76f"} Jul 09 18:58:19.220300 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:19.219996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:19.248768 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:19.248701 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" podStartSLOduration=1.869045546 podStartE2EDuration="6.248685392s" totalImagesPullingTime="3.270682146s" totalInitContainerRuntime="1.1089577s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:19.246867689 +0000 UTC m=+274.663582099" watchObservedRunningTime="2026-07-09 18:58:19.248685392 +0000 UTC m=+274.665399801" Jul 09 18:58:30.226257 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:30.226223 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-gm7r4" Jul 09 18:58:31.374235 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.374203 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9"] Jul 09 18:58:31.388223 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.388199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9"] Jul 09 18:58:31.388365 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.388292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.392537 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.392513 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 18:58:31.392678 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.392556 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 18:58:31.392678 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.392635 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w7f7f\"" Jul 09 18:58:31.392678 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.392650 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 18:58:31.483182 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.483142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e6b3768-700a-4a08-86f9-65925793a372-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.483364 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.483198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgf7\" (UniqueName: \"kubernetes.io/projected/8e6b3768-700a-4a08-86f9-65925793a372-kube-api-access-6jgf7\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.483364 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.483288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.483364 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.483328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.584361 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.584323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e6b3768-700a-4a08-86f9-65925793a372-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.584525 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.584370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgf7\" (UniqueName: \"kubernetes.io/projected/8e6b3768-700a-4a08-86f9-65925793a372-kube-api-access-6jgf7\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.584525 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.584399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.584525 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.584420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.585117 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.585094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8e6b3768-700a-4a08-86f9-65925793a372-manager-config\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.586903 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.586882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.586978 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.586950 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e6b3768-700a-4a08-86f9-65925793a372-metrics-cert\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.594333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.594310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgf7\" (UniqueName: \"kubernetes.io/projected/8e6b3768-700a-4a08-86f9-65925793a372-kube-api-access-6jgf7\") pod \"lws-controller-manager-9dfc7f7f4-dz7r9\" (UID: \"8e6b3768-700a-4a08-86f9-65925793a372\") " pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.697809 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.697781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:31.821645 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:31.821497 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9"] Jul 09 18:58:31.824205 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:58:31.824172 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6b3768_700a_4a08_86f9_65925793a372.slice/crio-2076b748d4f16ef72f76cd69399c24ad230c043edb553e3d7aea563607d1a188 WatchSource:0}: Error finding container 2076b748d4f16ef72f76cd69399c24ad230c043edb553e3d7aea563607d1a188: Status 404 returned error can't find the container with id 2076b748d4f16ef72f76cd69399c24ad230c043edb553e3d7aea563607d1a188 Jul 09 18:58:32.270916 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:32.270877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" event={"ID":"8e6b3768-700a-4a08-86f9-65925793a372","Type":"ContainerStarted","Data":"2076b748d4f16ef72f76cd69399c24ad230c043edb553e3d7aea563607d1a188"} Jul 09 18:58:33.279202 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:33.279043 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" event={"ID":"8e6b3768-700a-4a08-86f9-65925793a372","Type":"ContainerStarted","Data":"439474a039ac8c233cb671c57ab1942dfb0ae19438d5570063b9c85ba300aad9"} Jul 09 18:58:33.279202 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:33.279114 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:33.326170 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:33.326081 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" podStartSLOduration=1.19976889 podStartE2EDuration="2.326059319s" totalImagesPullingTime="1.126290429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:33.324305663 +0000 UTC m=+288.741020066" watchObservedRunningTime="2026-07-09 18:58:33.326059319 +0000 UTC m=+288.742773729" Jul 09 18:58:41.163257 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.163221 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt"] Jul 09 18:58:41.165432 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.165415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.168506 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.168486 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 18:58:41.168656 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.168553 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 18:58:41.169867 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.169847 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 18:58:41.170333 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.170318 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 18:58:41.171935 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.171907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c0afb1-990b-4773-a7f4-bfacd07d93d1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172050 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.171985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172397 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172397 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172397 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172397 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88lzs\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-kube-api-access-88lzs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172397 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.172646 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172520 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 18:58:41.172646 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172574 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 18:58:41.172758 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.172579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6b79v\"" Jul 09 18:58:41.189407 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.189384 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt"] Jul 09 18:58:41.272742 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88lzs\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-kube-api-access-88lzs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c0afb1-990b-4773-a7f4-bfacd07d93d1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.272939 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.272887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.273695 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.273666 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.275231 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.275202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.275332 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.275292 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.275332 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.275324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.275401 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.275330 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c0afb1-990b-4773-a7f4-bfacd07d93d1-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.294237 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.294216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.297800 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.297776 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88lzs\" (UniqueName: \"kubernetes.io/projected/48c0afb1-990b-4773-a7f4-bfacd07d93d1-kube-api-access-88lzs\") pod \"istiod-openshift-gateway-94bb8fbfd-h2fmt\" (UID: \"48c0afb1-990b-4773-a7f4-bfacd07d93d1\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.420111 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.420038 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d"] Jul 09 18:58:41.422866 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.422849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.425646 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.425624 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mhvkv\"" Jul 09 18:58:41.442354 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.442334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d"] Jul 09 18:58:41.474796 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.474770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:41.475168 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475250 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475250 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475339 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjb9\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-kube-api-access-7hjb9\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475339 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475436 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475436 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475436 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.475436 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.475431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576365 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjb9\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-kube-api-access-7hjb9\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.576856 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.576837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.577337 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.577177 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.577392 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.577332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.577446 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.577389 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.577446 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.577425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.579490 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.579459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.579653 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.579633 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.590392 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.590368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.591788 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.591768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjb9\" (UniqueName: \"kubernetes.io/projected/761dce66-2b21-4b62-8aa7-ed63a5aeda95-kube-api-access-7hjb9\") pod \"data-science-gateway-data-science-gateway-class-554689689d8rk8d\" (UID: \"761dce66-2b21-4b62-8aa7-ed63a5aeda95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.612613 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.612586 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt"] Jul 09 18:58:41.615919 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:58:41.615885 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c0afb1_990b_4773_a7f4_bfacd07d93d1.slice/crio-a927a3e619ca7acdefcd154beaed3ffa9fdb43cddd38a8330ae144fc87bf6b52 WatchSource:0}: Error finding container a927a3e619ca7acdefcd154beaed3ffa9fdb43cddd38a8330ae144fc87bf6b52: Status 404 returned error can't find the container with id a927a3e619ca7acdefcd154beaed3ffa9fdb43cddd38a8330ae144fc87bf6b52 Jul 09 18:58:41.735633 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.735555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:41.903182 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:41.903149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d"] Jul 09 18:58:41.908658 ip-10-0-131-3 kubenswrapper[2578]: W0709 18:58:41.908618 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761dce66_2b21_4b62_8aa7_ed63a5aeda95.slice/crio-edbac90df70c85d3d0af6a8c39810b25c32e3221f4c5fb560363171689bdafee WatchSource:0}: Error finding container edbac90df70c85d3d0af6a8c39810b25c32e3221f4c5fb560363171689bdafee: Status 404 returned error can't find the container with id edbac90df70c85d3d0af6a8c39810b25c32e3221f4c5fb560363171689bdafee Jul 09 18:58:42.321804 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:42.321759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" event={"ID":"761dce66-2b21-4b62-8aa7-ed63a5aeda95","Type":"ContainerStarted","Data":"edbac90df70c85d3d0af6a8c39810b25c32e3221f4c5fb560363171689bdafee"} Jul 09 18:58:42.323398 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:42.323369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" event={"ID":"48c0afb1-990b-4773-a7f4-bfacd07d93d1","Type":"ContainerStarted","Data":"a927a3e619ca7acdefcd154beaed3ffa9fdb43cddd38a8330ae144fc87bf6b52"} Jul 09 18:58:44.286181 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:44.286149 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9dfc7f7f4-dz7r9" Jul 09 18:58:45.210084 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.209560 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 18:58:45.210084 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.209642 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 18:58:45.212429 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.212394 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 18:58:45.212552 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.212466 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 18:58:45.212552 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.212504 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 18:58:45.226631 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.226609 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:58:45.226751 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.226614 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 18:58:45.232584 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.232564 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 18:58:45.341047 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.340997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" event={"ID":"48c0afb1-990b-4773-a7f4-bfacd07d93d1","Type":"ContainerStarted","Data":"e630e3e19ee4c9877478921d0020abbd044fce6f373d175bb2048438fe266ea3"} Jul 09 18:58:45.341486 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.341165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 18:58:45.343338 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.343313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" event={"ID":"761dce66-2b21-4b62-8aa7-ed63a5aeda95","Type":"ContainerStarted","Data":"1cb2e79ece4e9e82dcae0639791e6fe3100f0cadfe99b1926e621b4c7c78138a"} Jul 09 18:58:45.380748 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.380678 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" podStartSLOduration=0.789450992 podStartE2EDuration="4.380657247s" totalImagesPullingTime="3.591206255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:45.377587059 +0000 UTC m=+300.794301471" watchObservedRunningTime="2026-07-09 18:58:45.380657247 +0000 UTC m=+300.797371656" Jul 09 18:58:45.437917 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.437860 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" podStartSLOduration=1.13635325 podStartE2EDuration="4.437838742s" totalImagesPullingTime="3.301485492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:45.437107073 +0000 UTC m=+300.853821485" watchObservedRunningTime="2026-07-09 18:58:45.437838742 +0000 UTC m=+300.854553154" Jul 09 18:58:45.736334 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:45.736297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:46.349394 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:46.349359 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-h2fmt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 18:58:46.349783 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:46.349412 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" podUID="48c0afb1-990b-4773-a7f4-bfacd07d93d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:58:46.740368 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:46.740339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:47.352343 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:47.352307 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:47.353416 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:47.353396 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-554689689d8rk8d" Jul 09 18:58:49.348372 ip-10-0-131-3 kubenswrapper[2578]: I0709 18:58:49.348335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-h2fmt" Jul 09 19:00:31.512549 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.512516 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs"] Jul 09 19:00:31.515764 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.515748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.519203 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.519182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zq2wt\"" Jul 09 19:00:31.519288 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.519255 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 19:00:31.519288 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.519183 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 19:00:31.532871 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.532840 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs"] Jul 09 19:00:31.627494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.627459 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs"] Jul 09 19:00:31.627754 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:00:31.627719 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-mr5fz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" podUID="f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" Jul 09 19:00:31.687953 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.687923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.688128 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.687975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5fz\" (UniqueName: \"kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.785936 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.785854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.788752 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.788714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5fz\" (UniqueName: \"kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.788860 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.788819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.789262 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.789242 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.792410 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.792387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.814924 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.814893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5fz\" (UniqueName: \"kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:31.890039 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.890007 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume\") pod \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " Jul 09 19:00:31.890275 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.890251 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" (UID: "f43bbc34-afa6-4e30-a6f1-fcd0290a6d20"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:00:31.991082 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.991048 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr5fz\" (UniqueName: \"kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz\") pod \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\" (UID: \"f43bbc34-afa6-4e30-a6f1-fcd0290a6d20\") " Jul 09 19:00:31.991270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.991254 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-extensions-socket-volume\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:00:31.993113 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:31.993081 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz" (OuterVolumeSpecName: "kube-api-access-mr5fz") pod "f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" (UID: "f43bbc34-afa6-4e30-a6f1-fcd0290a6d20"). InnerVolumeSpecName "kube-api-access-mr5fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:00:32.092506 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:32.092435 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5fz\" (UniqueName: \"kubernetes.io/projected/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20-kube-api-access-mr5fz\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:00:32.109869 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:32.109832 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs"] Jul 09 19:00:32.111092 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:32.111069 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs"] Jul 09 19:00:32.789784 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:32.789751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:32.792693 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:00:32.792665 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" is forbidden: User \"system:node:ip-10-0-131-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-3.ec2.internal' and this object" podUID="f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:32.798397 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:00:32.798370 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4g9bs\" is forbidden: User \"system:node:ip-10-0-131-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-3.ec2.internal' and this object" podUID="f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4g9bs" Jul 09 19:00:33.105853 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:00:33.105757 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43bbc34-afa6-4e30-a6f1-fcd0290a6d20" path="/var/lib/kubelet/pods/f43bbc34-afa6-4e30-a6f1-fcd0290a6d20/volumes" Jul 09 19:01:20.126956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.126878 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg"] Jul 09 19:01:20.130359 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.130336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.133290 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.133273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-82xsr\"" Jul 09 19:01:20.149149 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.149123 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg"] Jul 09 19:01:20.202364 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202682 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202682 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202792 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.202832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.202808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkzj7\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-kube-api-access-hkzj7\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304231 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304231 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkzj7\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-kube-api-access-hkzj7\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304494 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304855 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-certs\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.304855 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.304828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-workload-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.305131 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.305047 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-data\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.305131 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.305057 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-credential-socket\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.305365 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.305163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.307022 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.306994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.307117 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.307026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.313148 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.313116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-istio-token\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.314013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.313992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkzj7\" (UniqueName: \"kubernetes.io/projected/5ad7303a-348c-43b4-be8f-f01aa6c912fb-kube-api-access-hkzj7\") pod \"maas-default-gateway-openshift-default-74949dcd89-kmrjg\" (UID: \"5ad7303a-348c-43b4-be8f-f01aa6c912fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.440452 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.440414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:20.578936 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.578909 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg"] Jul 09 19:01:20.580843 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:01:20.580813 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad7303a_348c_43b4_be8f_f01aa6c912fb.slice/crio-f36130bf974a066aa61b63ef0ea9f3f1130c85a3ce9bd57242dd847bf809f9d9 WatchSource:0}: Error finding container f36130bf974a066aa61b63ef0ea9f3f1130c85a3ce9bd57242dd847bf809f9d9: Status 404 returned error can't find the container with id f36130bf974a066aa61b63ef0ea9f3f1130c85a3ce9bd57242dd847bf809f9d9 Jul 09 19:01:20.583044 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.583011 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:01:20.583143 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.583080 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:01:20.583143 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.583109 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:01:20.990217 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.990179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" event={"ID":"5ad7303a-348c-43b4-be8f-f01aa6c912fb","Type":"ContainerStarted","Data":"049e43ccbe8efc955ca17e477798c7ecf86df659d59b1bca096a2a636b0eff37"} Jul 09 19:01:20.990407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:20.990243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" event={"ID":"5ad7303a-348c-43b4-be8f-f01aa6c912fb","Type":"ContainerStarted","Data":"f36130bf974a066aa61b63ef0ea9f3f1130c85a3ce9bd57242dd847bf809f9d9"} Jul 09 19:01:21.013724 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:21.013676 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podStartSLOduration=1.013657767 podStartE2EDuration="1.013657767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:21.012518145 +0000 UTC m=+456.429232555" watchObservedRunningTime="2026-07-09 19:01:21.013657767 +0000 UTC m=+456.430372177" Jul 09 19:01:21.441200 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:21.441165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:21.442448 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:21.442420 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:01:21.442560 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:21.442479 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:01:22.444986 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.444956 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:22.469559 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.469526 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:01:22.472977 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.472948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.476871 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.476843 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 19:01:22.477051 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.476942 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 19:01:22.477051 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.476986 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 19:01:22.477051 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.476995 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xcftl\"" Jul 09 19:01:22.485511 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.485477 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:01:22.516323 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.516289 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:01:22.625433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.625394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrddj\" (UniqueName: \"kubernetes.io/projected/f00f672c-4888-402a-973b-67cc103c8dcf-kube-api-access-vrddj\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.625623 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.625528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f00f672c-4888-402a-973b-67cc103c8dcf-config-file\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.726862 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.726766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrddj\" (UniqueName: \"kubernetes.io/projected/f00f672c-4888-402a-973b-67cc103c8dcf-kube-api-access-vrddj\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.727017 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.726892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f00f672c-4888-402a-973b-67cc103c8dcf-config-file\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.727538 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.727519 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f00f672c-4888-402a-973b-67cc103c8dcf-config-file\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.739459 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.739422 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrddj\" (UniqueName: \"kubernetes.io/projected/f00f672c-4888-402a-973b-67cc103c8dcf-kube-api-access-vrddj\") pod \"limitador-limitador-54d6d988d5-ddzsn\" (UID: \"f00f672c-4888-402a-973b-67cc103c8dcf\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.784075 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.784031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:22.921439 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.921410 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:01:22.923289 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:01:22.923259 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00f672c_4888_402a_973b_67cc103c8dcf.slice/crio-d75e3473b27df649a8a1675a288514b836a378bf4a68ce0a09cb6f4acb8070a3 WatchSource:0}: Error finding container d75e3473b27df649a8a1675a288514b836a378bf4a68ce0a09cb6f4acb8070a3: Status 404 returned error can't find the container with id d75e3473b27df649a8a1675a288514b836a378bf4a68ce0a09cb6f4acb8070a3 Jul 09 19:01:22.925290 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:22.925274 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:01:23.001863 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:23.001763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" event={"ID":"f00f672c-4888-402a-973b-67cc103c8dcf","Type":"ContainerStarted","Data":"d75e3473b27df649a8a1675a288514b836a378bf4a68ce0a09cb6f4acb8070a3"} Jul 09 19:01:23.002037 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:23.001988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:23.003051 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:23.003031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:01:26.018017 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:26.017976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" event={"ID":"f00f672c-4888-402a-973b-67cc103c8dcf","Type":"ContainerStarted","Data":"02113294e184284e9f882106dacc2f6696e67937b9bd5a4429a587d3947988c1"} Jul 09 19:01:26.018412 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:26.018125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:01:26.035973 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:26.035916 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" podStartSLOduration=1.370431357 podStartE2EDuration="4.035894348s" totalImagesPullingTime="2.665462991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:26.035535084 +0000 UTC m=+461.452249494" watchObservedRunningTime="2026-07-09 19:01:26.035894348 +0000 UTC m=+461.452608757" Jul 09 19:01:37.023711 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:01:37.023678 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ddzsn" Jul 09 19:02:37.150688 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:37.150602 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:02:44.360935 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.360901 2578 generic.go:356] "Generic (PLEG): container finished" podID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerID="049e43ccbe8efc955ca17e477798c7ecf86df659d59b1bca096a2a636b0eff37" exitCode=0 Jul 09 19:02:44.361348 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.360964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" event={"ID":"5ad7303a-348c-43b4-be8f-f01aa6c912fb","Type":"ContainerDied","Data":"049e43ccbe8efc955ca17e477798c7ecf86df659d59b1bca096a2a636b0eff37"} Jul 09 19:02:44.361348 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.361325 2578 scope.go:117] "RemoveContainer" containerID="049e43ccbe8efc955ca17e477798c7ecf86df659d59b1bca096a2a636b0eff37" Jul 09 19:02:44.362326 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.362295 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:02:44.362383 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.362369 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:02:44.362422 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:44.362403 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:02:45.367402 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:45.367314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" event={"ID":"5ad7303a-348c-43b4-be8f-f01aa6c912fb","Type":"ContainerStarted","Data":"2f1c5cb4889edeb9ba83d48ddf1b648c4bebd097345a1eecc381ff49327a6dae"} Jul 09 19:02:45.441363 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:45.441322 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:02:45.441636 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:45.441618 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:02:45.441687 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:45.441660 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:02:46.441233 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:46.441187 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:02:46.441675 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:46.441253 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:02:47.440983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:47.440945 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:02:47.441171 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:47.441006 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:02:48.441187 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:48.441145 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:02:48.441588 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:48.441209 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:02:49.440884 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:49.440849 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-74949dcd89-kmrjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 09 19:02:49.441061 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:49.440916 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" podUID="5ad7303a-348c-43b4-be8f-f01aa6c912fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 09 19:02:50.445010 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:50.444979 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:02:50.445479 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:50.445249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:02:50.446082 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:02:50.446066 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74949dcd89-kmrjg" Jul 09 19:03:04.930285 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:04.930255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:03:15.827500 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:15.827455 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:03:38.234048 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:38.234010 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:03:40.339406 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.339370 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2"] Jul 09 19:03:40.343304 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.343277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.351905 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.348056 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l6hgx\"" Jul 09 19:03:40.351905 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.348310 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 19:03:40.351905 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.348374 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 19:03:40.351905 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.349112 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 19:03:40.358741 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.358697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2"] Jul 09 19:03:40.418743 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6gf\" (UniqueName: \"kubernetes.io/projected/a0531612-9be8-421b-92c6-321cf7d8a50f-kube-api-access-nk6gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.418932 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.418932 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.418932 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0531612-9be8-421b-92c6-321cf7d8a50f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.418932 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.418932 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.418904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520012 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.519976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520012 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520229 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0531612-9be8-421b-92c6-321cf7d8a50f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520266 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520319 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520403 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520381 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520475 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6gf\" (UniqueName: \"kubernetes.io/projected/a0531612-9be8-421b-92c6-321cf7d8a50f-kube-api-access-nk6gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520565 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.520636 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.520617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.522425 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.522402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0531612-9be8-421b-92c6-321cf7d8a50f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.522595 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.522578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0531612-9be8-421b-92c6-321cf7d8a50f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.528294 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.528270 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6gf\" (UniqueName: \"kubernetes.io/projected/a0531612-9be8-421b-92c6-321cf7d8a50f-kube-api-access-nk6gf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2\" (UID: \"a0531612-9be8-421b-92c6-321cf7d8a50f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.658913 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.658866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:40.788804 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:40.788776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2"] Jul 09 19:03:40.790835 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:03:40.790800 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0531612_9be8_421b_92c6_321cf7d8a50f.slice/crio-5ab23e431cad578ab3f84555ae60bb740aa50e90c1c94ce9de9750523264615e WatchSource:0}: Error finding container 5ab23e431cad578ab3f84555ae60bb740aa50e90c1c94ce9de9750523264615e: Status 404 returned error can't find the container with id 5ab23e431cad578ab3f84555ae60bb740aa50e90c1c94ce9de9750523264615e Jul 09 19:03:41.620446 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:41.620403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" event={"ID":"a0531612-9be8-421b-92c6-321cf7d8a50f","Type":"ContainerStarted","Data":"5ab23e431cad578ab3f84555ae60bb740aa50e90c1c94ce9de9750523264615e"} Jul 09 19:03:42.532273 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:42.532234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:03:42.626354 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:42.626315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" event={"ID":"a0531612-9be8-421b-92c6-321cf7d8a50f","Type":"ContainerStarted","Data":"636d34539131bf0a6b4fa063924d385dd790940b136435536641414b900cb927"} Jul 09 19:03:42.650080 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:42.650031 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" podStartSLOduration=1.562482589 podStartE2EDuration="2.650015592s" totalImagesPullingTime="1.087533003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:42.647930985 +0000 UTC m=+598.064645394" watchObservedRunningTime="2026-07-09 19:03:42.650015592 +0000 UTC m=+598.066730002" Jul 09 19:03:45.255203 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:45.255108 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:03:45.255203 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:45.255158 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:03:50.659293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:50.659262 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:50.659967 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:50.659365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:50.671981 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:50.671950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:51.677682 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:51.677646 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2" Jul 09 19:03:57.724686 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:03:57.724645 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:04:14.835401 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:04:14.835314 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:05:13.960620 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:05:13.960581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:05:23.754907 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:05:23.754873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:05:33.455996 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:05:33.455956 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:05:43.224539 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:05:43.224449 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:05:53.832911 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:05:53.832873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:06:21.438442 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:06:21.438404 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:06:35.934351 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:06:35.934312 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:09.928397 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:09.928306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:19.631694 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:19.631657 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:30.329858 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:30.329823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:36.227693 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:36.227647 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:47.144629 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:47.144595 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:07:57.228827 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:07:57.228784 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:08:08.227365 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:08.227329 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:08:18.423892 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:18.423857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:08:40.133125 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:40.133039 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:08:45.280723 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:45.280696 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:08:45.281522 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:45.281502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:08:51.933278 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:08:51.933244 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:09:16.236627 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:09:16.236586 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:09:26.731496 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:09:26.731457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:09:42.033026 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:09:42.032988 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:09:51.432817 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:09:51.432780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:10:02.040646 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:10:02.040601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:10:12.028188 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:10:12.028088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:10:21.330225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:10:21.330183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:12:08.432188 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:12:08.432153 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:12:19.836138 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:12:19.836101 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:12:28.545172 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:12:28.545133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:12:40.552781 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:12:40.552746 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:12:51.044832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:12:51.044795 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:01.853573 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:01.853534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:11.134689 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:11.134604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:21.924569 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:21.924531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:33.631440 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:33.631409 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:42.026566 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:42.026523 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:13:45.306259 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:45.306231 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:13:45.308648 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:45.308627 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:13:52.535459 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:13:52.535428 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:14:14.631042 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:14:14.631004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:14:25.028490 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:14:25.028454 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:14:34.832218 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:14:34.832183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:14:44.727654 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:14:44.727564 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:14:55.139269 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:14:55.139232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:15:04.336003 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:15:04.335965 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:15:17.643632 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:15:17.643594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:15:26.826478 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:15:26.826444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:15:34.951267 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:15:34.951235 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:15:44.755843 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:15:44.755811 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:02.941366 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:02.941329 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:12.831581 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:12.831472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:22.632971 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:22.632937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:32.532818 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:32.532780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:41.936673 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:41.936637 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:16:54.529552 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:16:54.529517 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:02.438582 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:02.438547 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:14.529963 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:14.529927 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:26.334716 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:26.334670 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:37.626033 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:37.625948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:50.329396 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:50.329359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:17:59.437339 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:17:59.437303 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:09.633978 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.633941 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:09.807987 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.807950 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:09.811296 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.811274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.814200 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.814180 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-14bca9d0-openshift-default-dockercfg-jqp4m\"" Jul 09 19:18:09.841173 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.841148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:09.988116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtwg\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:09.988384 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:09.988307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089774 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtwg\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.089929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.089948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090406 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090406 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.090635 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.090617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.092161 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.092132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.092398 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.092377 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.099327 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.099300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtwg\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.099429 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.099329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token\") pod \"e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.123032 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.123009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:10.255350 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.255325 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:10.257064 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:18:10.257033 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adbda19_6145_4083_9783_8430d6567849.slice/crio-c8c1457f00a25c6fda1e1d2336c1f15c91dbea3a73b13364c384f0d86a3ac02b WatchSource:0}: Error finding container c8c1457f00a25c6fda1e1d2336c1f15c91dbea3a73b13364c384f0d86a3ac02b: Status 404 returned error can't find the container with id c8c1457f00a25c6fda1e1d2336c1f15c91dbea3a73b13364c384f0d86a3ac02b Jul 09 19:18:10.259348 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.259316 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:10.259435 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.259379 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:10.259435 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.259407 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:10.430472 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.430438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" event={"ID":"8adbda19-6145-4083-9783-8430d6567849","Type":"ContainerStarted","Data":"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e"} Jul 09 19:18:10.430623 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.430504 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" event={"ID":"8adbda19-6145-4083-9783-8430d6567849","Type":"ContainerStarted","Data":"c8c1457f00a25c6fda1e1d2336c1f15c91dbea3a73b13364c384f0d86a3ac02b"} Jul 09 19:18:10.455389 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:10.455342 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" podStartSLOduration=1.45532357 podStartE2EDuration="1.45532357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:10.454983049 +0000 UTC m=+1465.871697460" watchObservedRunningTime="2026-07-09 19:18:10.45532357 +0000 UTC m=+1465.872037980" Jul 09 19:18:11.123556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:11.123517 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:11.128497 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:11.128475 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:11.434776 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:11.434719 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:11.435692 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:11.435670 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:16.230808 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:16.230777 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:16.231267 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:16.230957 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" podUID="8adbda19-6145-4083-9783-8430d6567849" containerName="istio-proxy" containerID="cri-o://1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e" gracePeriod=30 Jul 09 19:18:22.681772 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.681743 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:22.710904 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.710877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.710912 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.710934 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjtwg\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.710964 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.710991 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711046 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711071 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711385 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711107 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711385 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711163 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data\") pod \"8adbda19-6145-4083-9783-8430d6567849\" (UID: \"8adbda19-6145-4083-9783-8430d6567849\") " Jul 09 19:18:22.711385 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:22.711561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711434 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.711561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711427 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:22.711561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711435 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:22.711561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.711537 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:18:22.712264 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.712236 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data" (OuterVolumeSpecName: "istio-data") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:22.713634 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.713595 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg" (OuterVolumeSpecName: "kube-api-access-tjtwg") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "kube-api-access-tjtwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:22.713634 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.713611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token" (OuterVolumeSpecName: "istio-token") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:22.713831 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.713669 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:22.713831 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.713677 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8adbda19-6145-4083-9783-8430d6567849" (UID: "8adbda19-6145-4083-9783-8430d6567849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:18:22.809832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.809724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:22.810187 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.810169 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adbda19-6145-4083-9783-8430d6567849" containerName="istio-proxy" Jul 09 19:18:22.810242 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.810189 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adbda19-6145-4083-9783-8430d6567849" containerName="istio-proxy" Jul 09 19:18:22.810289 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.810254 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adbda19-6145-4083-9783-8430d6567849" containerName="istio-proxy" Jul 09 19:18:22.812124 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812095 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8adbda19-6145-4083-9783-8430d6567849-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812124 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812125 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812139 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812152 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812164 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812176 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjtwg\" (UniqueName: \"kubernetes.io/projected/8adbda19-6145-4083-9783-8430d6567849-kube-api-access-tjtwg\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812191 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8adbda19-6145-4083-9783-8430d6567849-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.812324 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.812204 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8adbda19-6145-4083-9783-8430d6567849-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:22.813444 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.813424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.826901 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.826427 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6ddac54e-openshift-default-dockercfg-dbqcr\"" Jul 09 19:18:22.832081 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.832054 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:22.913506 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913506 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913710 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913710 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913710 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrwr\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913838 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913838 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913838 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:22.913933 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:22.913856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.014741 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrwr\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.014909 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.014909 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.014909 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015064 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015064 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015064 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.014984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015064 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015064 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015372 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015342 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015513 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015513 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015451 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015619 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015529 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.015815 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.015800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.017344 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.017327 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.017562 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.017541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.027075 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.027051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.027490 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.027473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrwr\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr\") pod \"e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.132626 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.132597 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:23.268965 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.268938 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:23.270758 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:18:23.270715 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1fecdc_8c96_427e_be20_7bf244d6741a.slice/crio-826e6e589d0f67e0a869594e3eb5efa7d2e9993dde7f8add86ad8df268c02921 WatchSource:0}: Error finding container 826e6e589d0f67e0a869594e3eb5efa7d2e9993dde7f8add86ad8df268c02921: Status 404 returned error can't find the container with id 826e6e589d0f67e0a869594e3eb5efa7d2e9993dde7f8add86ad8df268c02921 Jul 09 19:18:23.273074 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.273040 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:23.273173 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.273114 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:23.273173 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.273157 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:23.503747 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.503698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" event={"ID":"ee1fecdc-8c96-427e-be20-7bf244d6741a","Type":"ContainerStarted","Data":"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449"} Jul 09 19:18:23.503939 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.503769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" event={"ID":"ee1fecdc-8c96-427e-be20-7bf244d6741a","Type":"ContainerStarted","Data":"826e6e589d0f67e0a869594e3eb5efa7d2e9993dde7f8add86ad8df268c02921"} Jul 09 19:18:23.505222 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.505194 2578 generic.go:356] "Generic (PLEG): container finished" podID="8adbda19-6145-4083-9783-8430d6567849" containerID="1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e" exitCode=0 Jul 09 19:18:23.505349 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.505232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" event={"ID":"8adbda19-6145-4083-9783-8430d6567849","Type":"ContainerDied","Data":"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e"} Jul 09 19:18:23.505349 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.505240 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" Jul 09 19:18:23.505349 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.505271 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n" event={"ID":"8adbda19-6145-4083-9783-8430d6567849","Type":"ContainerDied","Data":"c8c1457f00a25c6fda1e1d2336c1f15c91dbea3a73b13364c384f0d86a3ac02b"} Jul 09 19:18:23.505349 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.505300 2578 scope.go:117] "RemoveContainer" containerID="1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e" Jul 09 19:18:23.524146 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.524121 2578 scope.go:117] "RemoveContainer" containerID="1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e" Jul 09 19:18:23.524434 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:18:23.524412 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e\": container with ID starting with 1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e not found: ID does not exist" containerID="1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e" Jul 09 19:18:23.524513 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.524443 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e"} err="failed to get container status \"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e\": rpc error: code = NotFound desc = could not find container \"1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e\": container with ID starting with 1180a4af0fe2444d7265264008102d7d627231e118c880dea9fe23284b79ad5e not found: ID does not exist" Jul 09 19:18:23.528139 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.528083 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" podStartSLOduration=1.528066027 podStartE2EDuration="1.528066027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:23.526718771 +0000 UTC m=+1478.943433181" watchObservedRunningTime="2026-07-09 19:18:23.528066027 +0000 UTC m=+1478.944780438" Jul 09 19:18:23.542029 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.542005 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:23.547703 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:23.547680 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-14bca9d0-openshift-default-684fff48b7-2nl5n"] Jul 09 19:18:24.133643 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:24.133613 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:24.138519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:24.138494 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:24.511347 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:24.511252 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:24.512192 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:24.512172 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:25.107062 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:25.107027 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adbda19-6145-4083-9783-8430d6567849" path="/var/lib/kubelet/pods/8adbda19-6145-4083-9783-8430d6567849/volumes" Jul 09 19:18:27.026545 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:27.026510 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:35.179355 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.179316 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:35.179826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.179520 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" podUID="ee1fecdc-8c96-427e-be20-7bf244d6741a" containerName="istio-proxy" containerID="cri-o://a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449" gracePeriod=30 Jul 09 19:18:35.312574 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.312548 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:35.439046 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.438972 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439046 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439022 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439061 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439110 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdrwr\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439164 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439188 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439232 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439497 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439267 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439497 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439290 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy\") pod \"ee1fecdc-8c96-427e-be20-7bf244d6741a\" (UID: \"ee1fecdc-8c96-427e-be20-7bf244d6741a\") " Jul 09 19:18:35.439497 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439392 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:35.439621 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439591 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.439998 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.439967 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:35.441203 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.441178 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:18:35.441271 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.441217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:35.442030 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.441998 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:35.442030 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.442014 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:35.442213 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.442032 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:18:35.443432 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.443400 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr" (OuterVolumeSpecName: "kube-api-access-xdrwr") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "kube-api-access-xdrwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:35.443590 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.443574 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee1fecdc-8c96-427e-be20-7bf244d6741a" (UID: "ee1fecdc-8c96-427e-be20-7bf244d6741a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:35.540876 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540844 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.540876 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540871 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrwr\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-kube-api-access-xdrwr\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.540876 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540883 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.541085 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540892 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.541085 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540900 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fecdc-8c96-427e-be20-7bf244d6741a-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.541085 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540908 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.541085 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540916 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.541085 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.540923 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fecdc-8c96-427e-be20-7bf244d6741a-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:35.559658 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.559622 2578 generic.go:356] "Generic (PLEG): container finished" podID="ee1fecdc-8c96-427e-be20-7bf244d6741a" containerID="a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449" exitCode=0 Jul 09 19:18:35.559797 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.559668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" event={"ID":"ee1fecdc-8c96-427e-be20-7bf244d6741a","Type":"ContainerDied","Data":"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449"} Jul 09 19:18:35.559797 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.559673 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" Jul 09 19:18:35.559797 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.559738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9" event={"ID":"ee1fecdc-8c96-427e-be20-7bf244d6741a","Type":"ContainerDied","Data":"826e6e589d0f67e0a869594e3eb5efa7d2e9993dde7f8add86ad8df268c02921"} Jul 09 19:18:35.559797 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.559762 2578 scope.go:117] "RemoveContainer" containerID="a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449" Jul 09 19:18:35.568185 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.568167 2578 scope.go:117] "RemoveContainer" containerID="a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449" Jul 09 19:18:35.568420 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:18:35.568403 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449\": container with ID starting with a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449 not found: ID does not exist" containerID="a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449" Jul 09 19:18:35.568467 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.568428 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449"} err="failed to get container status \"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449\": rpc error: code = NotFound desc = could not find container \"a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449\": container with ID starting with a2bfafcb94b986afabb3d0793a5ac0e0400b392d4b4b7908518905a567381449 not found: ID does not exist" Jul 09 19:18:35.583265 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.583239 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:35.587363 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:35.587339 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6ddac54e-openshift-default-5cfdb88567-vnmz9"] Jul 09 19:18:37.105855 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:37.105813 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1fecdc-8c96-427e-be20-7bf244d6741a" path="/var/lib/kubelet/pods/ee1fecdc-8c96-427e-be20-7bf244d6741a/volumes" Jul 09 19:18:37.336601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:37.336562 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:41.664471 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.664439 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:41.664968 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.664897 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1fecdc-8c96-427e-be20-7bf244d6741a" containerName="istio-proxy" Jul 09 19:18:41.664968 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.664916 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1fecdc-8c96-427e-be20-7bf244d6741a" containerName="istio-proxy" Jul 09 19:18:41.665047 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.664980 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1fecdc-8c96-427e-be20-7bf244d6741a" containerName="istio-proxy" Jul 09 19:18:41.669674 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.669651 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.672628 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.672599 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2d937836-openshift-default-dockercfg-8nl56\"" Jul 09 19:18:41.686317 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.686292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:41.799892 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.799855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.799892 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.799895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrkc\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.799930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.799986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.800030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.800046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.800064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800251 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.800110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.800251 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.800128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901217 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901217 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrkc\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901639 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901639 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901802 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901780 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901892 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901943 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.901978 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.901949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.902194 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.902174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.903664 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.903639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.903965 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.903945 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.910505 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.910476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.910964 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.910936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrkc\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc\") pod \"e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:41.980125 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:41.980036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:42.152776 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.152640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:42.155650 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:18:42.155623 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6272c0fb_244f_47f8_a8d5_488b9bf87d26.slice/crio-dbb8782394019220b295d3c8ea0d2c99252daa9c9ec8ab0f3506863db81cd360 WatchSource:0}: Error finding container dbb8782394019220b295d3c8ea0d2c99252daa9c9ec8ab0f3506863db81cd360: Status 404 returned error can't find the container with id dbb8782394019220b295d3c8ea0d2c99252daa9c9ec8ab0f3506863db81cd360 Jul 09 19:18:42.157698 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.157665 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:42.157811 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.157756 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:42.157811 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.157787 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:18:42.598612 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.598572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" event={"ID":"6272c0fb-244f-47f8-a8d5-488b9bf87d26","Type":"ContainerStarted","Data":"9dae9ac28f66918a2894ef55578649521dea12fff070a7b0a70dca2fdb59773d"} Jul 09 19:18:42.598810 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.598672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" event={"ID":"6272c0fb-244f-47f8-a8d5-488b9bf87d26","Type":"ContainerStarted","Data":"dbb8782394019220b295d3c8ea0d2c99252daa9c9ec8ab0f3506863db81cd360"} Jul 09 19:18:42.621781 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.621710 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" podStartSLOduration=1.621691052 podStartE2EDuration="1.621691052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:42.619976716 +0000 UTC m=+1498.036691159" watchObservedRunningTime="2026-07-09 19:18:42.621691052 +0000 UTC m=+1498.038405463" Jul 09 19:18:42.980723 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:42.980640 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:43.985337 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:43.985306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:44.609438 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:44.609410 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:44.610394 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:44.610369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:45.333362 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:45.333323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:18:45.336603 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:45.336582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:18:45.936918 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:45.936886 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:47.432671 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:47.432631 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:47.624235 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:47.624195 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" podUID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" containerName="istio-proxy" containerID="cri-o://9dae9ac28f66918a2894ef55578649521dea12fff070a7b0a70dca2fdb59773d" gracePeriod=30 Jul 09 19:18:53.653794 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.653763 2578 generic.go:356] "Generic (PLEG): container finished" podID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" containerID="9dae9ac28f66918a2894ef55578649521dea12fff070a7b0a70dca2fdb59773d" exitCode=0 Jul 09 19:18:53.654142 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.653835 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" event={"ID":"6272c0fb-244f-47f8-a8d5-488b9bf87d26","Type":"ContainerDied","Data":"9dae9ac28f66918a2894ef55578649521dea12fff070a7b0a70dca2fdb59773d"} Jul 09 19:18:53.773585 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.773560 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:53.920600 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920570 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmrkc\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920600 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920624 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920768 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920813 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920954 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920954 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920887 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920954 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920909 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.920954 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:53.920954 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.920926 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data\") pod \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\" (UID: \"6272c0fb-244f-47f8-a8d5-488b9bf87d26\") " Jul 09 19:18:53.921199 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:53.921199 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921158 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:53.921308 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921278 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:18:53.921308 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921294 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:53.921420 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921316 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:53.921420 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921331 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:53.921928 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.921897 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data" (OuterVolumeSpecName: "istio-data") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:53.922980 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.922940 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc" (OuterVolumeSpecName: "kube-api-access-tmrkc") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "kube-api-access-tmrkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:53.923215 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.923187 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:18:53.923215 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.923190 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token" (OuterVolumeSpecName: "istio-token") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:18:53.923321 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:53.923248 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6272c0fb-244f-47f8-a8d5-488b9bf87d26" (UID: "6272c0fb-244f-47f8-a8d5-488b9bf87d26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:18:54.021965 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021938 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.021965 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021961 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.021965 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021971 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.022178 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021979 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.022178 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021987 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrkc\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-kube-api-access-tmrkc\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.022178 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.021994 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6272c0fb-244f-47f8-a8d5-488b9bf87d26-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:18:54.658901 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.658864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" event={"ID":"6272c0fb-244f-47f8-a8d5-488b9bf87d26","Type":"ContainerDied","Data":"dbb8782394019220b295d3c8ea0d2c99252daa9c9ec8ab0f3506863db81cd360"} Jul 09 19:18:54.658901 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.658890 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q" Jul 09 19:18:54.659454 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.658925 2578 scope.go:117] "RemoveContainer" containerID="9dae9ac28f66918a2894ef55578649521dea12fff070a7b0a70dca2fdb59773d" Jul 09 19:18:54.681106 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.681079 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:54.685111 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.685088 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2d937836-openshift-default-5f4787b48c-9m46q"] Jul 09 19:18:54.824433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:54.824404 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:18:55.107397 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:18:55.107315 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" path="/var/lib/kubelet/pods/6272c0fb-244f-47f8-a8d5-488b9bf87d26/volumes" Jul 09 19:20:17.238602 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:17.238568 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:20:25.035921 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:25.035882 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:20:31.525877 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.525845 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:20:31.526278 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.526171 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" containerName="istio-proxy" Jul 09 19:20:31.526278 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.526182 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" containerName="istio-proxy" Jul 09 19:20:31.526278 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.526247 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6272c0fb-244f-47f8-a8d5-488b9bf87d26" containerName="istio-proxy" Jul 09 19:20:31.529255 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.529237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.532094 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.532073 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-87c27070-openshift-default-dockercfg-2h2qd\"" Jul 09 19:20:31.552373 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.552348 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:20:31.647942 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.647907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648119 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.647956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648119 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648197 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648197 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648197 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648308 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648308 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.648308 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.648272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9tp8\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749089 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749089 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9tp8\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749597 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749663 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749747 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.749874 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.749858 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.750150 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.750125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.750258 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.750241 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.751531 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.751514 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.751698 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.751683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.757371 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.757350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.757462 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.757418 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9tp8\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8\") pod \"e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.839013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.838928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:31.972643 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.972616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:20:31.975310 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:20:31.975283 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838381c3_a6ed_4d13_86d3_a6cc69256749.slice/crio-cf8ed56b2e2ffcb86b11af5ff834f886e2361b29f38efd2bc9b2525812e50388 WatchSource:0}: Error finding container cf8ed56b2e2ffcb86b11af5ff834f886e2361b29f38efd2bc9b2525812e50388: Status 404 returned error can't find the container with id cf8ed56b2e2ffcb86b11af5ff834f886e2361b29f38efd2bc9b2525812e50388 Jul 09 19:20:31.977417 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.977379 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:31.977503 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.977469 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:31.977540 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:31.977509 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:32.099403 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:32.099318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerStarted","Data":"a85b63cea73d54682ab3511b9b8bd4dad56c98f18f94eaec3571c403ac5065cb"} Jul 09 19:20:32.099403 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:32.099374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerStarted","Data":"cf8ed56b2e2ffcb86b11af5ff834f886e2361b29f38efd2bc9b2525812e50388"} Jul 09 19:20:32.122542 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:32.122480 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podStartSLOduration=1.122459609 podStartE2EDuration="1.122459609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:20:32.119447859 +0000 UTC m=+1607.536162272" watchObservedRunningTime="2026-07-09 19:20:32.122459609 +0000 UTC m=+1607.539174001" Jul 09 19:20:32.838997 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:32.838966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:33.844143 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:33.844113 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:34.111038 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:34.110952 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:34.111999 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:34.111981 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:34.635671 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:34.635641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:20:45.037872 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:45.037821 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:20:49.112868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.112761 2578 patch_prober.go:28] interesting pod/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 09 19:20:49.112868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.112819 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 09 19:20:49.184467 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.184431 2578 generic.go:356] "Generic (PLEG): container finished" podID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerID="a85b63cea73d54682ab3511b9b8bd4dad56c98f18f94eaec3571c403ac5065cb" exitCode=0 Jul 09 19:20:49.184618 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.184505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerDied","Data":"a85b63cea73d54682ab3511b9b8bd4dad56c98f18f94eaec3571c403ac5065cb"} Jul 09 19:20:49.184945 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.184930 2578 scope.go:117] "RemoveContainer" containerID="a85b63cea73d54682ab3511b9b8bd4dad56c98f18f94eaec3571c403ac5065cb" Jul 09 19:20:49.185983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.185953 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:49.186049 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.186026 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:49.186084 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:49.186059 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:20:50.192263 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:50.192222 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerStarted","Data":"d12bd37a1a8daf027883d335d2e52e3891cfcf491f36d316a2f09a3cab50fc05"} Jul 09 19:20:50.840006 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:50.839965 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:50.840221 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:50.840197 2578 patch_prober.go:28] interesting pod/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 09 19:20:50.840283 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:50.840251 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 09 19:20:51.839937 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:51.839891 2578 patch_prober.go:28] interesting pod/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 09 19:20:51.840396 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:51.839967 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 09 19:20:52.839998 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:52.839962 2578 patch_prober.go:28] interesting pod/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Jul 09 19:20:52.840376 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:52.840020 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Jul 09 19:20:53.409498 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:53.409461 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:20:53.409719 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:53.409694 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" containerID="cri-o://d12bd37a1a8daf027883d335d2e52e3891cfcf491f36d316a2f09a3cab50fc05" gracePeriod=30 Jul 09 19:20:53.540806 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:53.540771 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:20:59.233127 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.233091 2578 generic.go:356] "Generic (PLEG): container finished" podID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerID="d12bd37a1a8daf027883d335d2e52e3891cfcf491f36d316a2f09a3cab50fc05" exitCode=0 Jul 09 19:20:59.233580 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.233165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerDied","Data":"d12bd37a1a8daf027883d335d2e52e3891cfcf491f36d316a2f09a3cab50fc05"} Jul 09 19:20:59.233580 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.233234 2578 scope.go:117] "RemoveContainer" containerID="a85b63cea73d54682ab3511b9b8bd4dad56c98f18f94eaec3571c403ac5065cb" Jul 09 19:20:59.261440 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.261419 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:20:59.415946 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.415907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.415962 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416033 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416116 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416315 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416132 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416315 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416315 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416315 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9tp8\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8\") pod \"838381c3-a6ed-4d13-86d3-a6cc69256749\" (UID: \"838381c3-a6ed-4d13-86d3-a6cc69256749\") " Jul 09 19:20:59.416518 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416315 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:20:59.416518 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416401 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:20:59.416518 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416421 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:20:59.416660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416508 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:20:59.416660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416592 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/838381c3-a6ed-4d13-86d3-a6cc69256749-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.416660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416612 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.416660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416625 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.416660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416639 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.417059 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.416817 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data" (OuterVolumeSpecName: "istio-data") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:20:59.418311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.418274 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:20:59.418311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.418302 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8" (OuterVolumeSpecName: "kube-api-access-j9tp8") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "kube-api-access-j9tp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:20:59.418459 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.418330 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:20:59.418459 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.418379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token" (OuterVolumeSpecName: "istio-token") pod "838381c3-a6ed-4d13-86d3-a6cc69256749" (UID: "838381c3-a6ed-4d13-86d3-a6cc69256749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:20:59.517639 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.517610 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.517639 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.517634 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.517639 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.517644 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9tp8\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-kube-api-access-j9tp8\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.517848 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.517654 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:20:59.517848 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:20:59.517668 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/838381c3-a6ed-4d13-86d3-a6cc69256749-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:21:00.238279 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:00.238247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" event={"ID":"838381c3-a6ed-4d13-86d3-a6cc69256749","Type":"ContainerDied","Data":"cf8ed56b2e2ffcb86b11af5ff834f886e2361b29f38efd2bc9b2525812e50388"} Jul 09 19:21:00.238721 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:00.238287 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h" Jul 09 19:21:00.238721 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:00.238304 2578 scope.go:117] "RemoveContainer" containerID="d12bd37a1a8daf027883d335d2e52e3891cfcf491f36d316a2f09a3cab50fc05" Jul 09 19:21:00.294448 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:00.294411 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:21:00.299025 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:00.298996 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-87c27070-openshift-default-7c8f4cc947-2t65h"] Jul 09 19:21:01.105798 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:01.105769 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" path="/var/lib/kubelet/pods/838381c3-a6ed-4d13-86d3-a6cc69256749/volumes" Jul 09 19:21:13.616487 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.616426 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:21:13.616986 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.616935 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:21:13.616986 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.616955 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:21:13.617105 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.617074 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:21:13.619962 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.619937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.622785 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.622760 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bb59c5dc-openshift-default-dockercfg-v4qjc\"" Jul 09 19:21:13.635251 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.635229 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:21:13.740337 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740669 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740669 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qz9x\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740669 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.740669 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.740635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.841837 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.841803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.841852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qz9x\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.841896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.841920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.841963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842596 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842596 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842596 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842320 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842596 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.842832 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.842795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.844278 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.844257 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.844377 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.844305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.851304 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.851283 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.851711 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.851690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qz9x\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x\") pod \"e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:13.930701 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:13.930667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:14.069000 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.068817 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:21:14.071714 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:21:14.071682 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf1e6ff_3dd5_4b7b_bbd9_615d7d3ee700.slice/crio-92264877f5e8461c05e71f3da6c60cf4c86511fc7188c7b934fc6404a86414a8 WatchSource:0}: Error finding container 92264877f5e8461c05e71f3da6c60cf4c86511fc7188c7b934fc6404a86414a8: Status 404 returned error can't find the container with id 92264877f5e8461c05e71f3da6c60cf4c86511fc7188c7b934fc6404a86414a8 Jul 09 19:21:14.073831 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.073797 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:14.073911 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.073862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:14.073911 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.073893 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:14.300868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.300724 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerStarted","Data":"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba"} Jul 09 19:21:14.300868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.300855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerStarted","Data":"92264877f5e8461c05e71f3da6c60cf4c86511fc7188c7b934fc6404a86414a8"} Jul 09 19:21:14.320590 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.320540 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podStartSLOduration=1.320523572 podStartE2EDuration="1.320523572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:21:14.319036485 +0000 UTC m=+1649.735750897" watchObservedRunningTime="2026-07-09 19:21:14.320523572 +0000 UTC m=+1649.737237981" Jul 09 19:21:14.931573 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:14.931535 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:15.936567 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:15.936530 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:16.310447 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:16.310366 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:16.311297 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:16.311276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:18.930181 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:18.930150 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:21:27.368943 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.368911 2578 generic.go:356] "Generic (PLEG): container finished" podID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerID="9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba" exitCode=0 Jul 09 19:21:27.369367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.368985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerDied","Data":"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba"} Jul 09 19:21:27.369367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.369351 2578 scope.go:117] "RemoveContainer" containerID="9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba" Jul 09 19:21:27.370342 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.370314 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:27.370421 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.370398 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:27.370553 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:27.370443 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:21:28.377533 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:28.377497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerStarted","Data":"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32"} Jul 09 19:21:28.931486 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:28.931433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:28.932580 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:28.932550 2578 patch_prober.go:28] interesting pod/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 09 19:21:28.932697 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:28.932610 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 09 19:21:29.931418 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:29.931381 2578 patch_prober.go:28] interesting pod/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 09 19:21:29.931849 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:29.931460 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 09 19:21:30.931570 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:30.931529 2578 patch_prober.go:28] interesting pod/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 09 19:21:30.932007 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:30.931590 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 09 19:21:31.310924 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:31.310826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:31.932009 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:31.931973 2578 patch_prober.go:28] interesting pod/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 09 19:21:31.932431 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:31.932028 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 09 19:21:32.934984 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:32.934950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:32.936088 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:32.936063 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:21:41.026359 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:41.026324 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:21:56.029121 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:21:56.029086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:22:06.736868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:06.736786 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:22:48.732527 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:48.732489 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:22:52.801692 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:52.801652 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:22:52.802145 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:52.801881 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" containerID="cri-o://09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32" gracePeriod=30 Jul 09 19:22:57.229929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:57.229892 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:22:58.049282 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.049256 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:22:58.069205 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069205 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069192 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qz9x\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069381 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069221 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069381 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069237 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069381 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069283 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069381 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069368 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069406 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069435 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069450 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:58.069601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069481 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket\") pod \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\" (UID: \"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700\") " Jul 09 19:22:58.069601 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069561 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:58.069899 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069825 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.069899 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:58.069899 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.069845 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.070358 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.070326 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:58.070358 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.070346 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:22:58.071537 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.071517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x" (OuterVolumeSpecName: "kube-api-access-2qz9x") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "kube-api-access-2qz9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:22:58.071767 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.071581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:22:58.071820 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.071771 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:22:58.071882 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.071863 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" (UID: "ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:22:58.171019 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.170968 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qz9x\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-kube-api-access-2qz9x\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171019 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.170997 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171019 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.171010 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171019 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.171021 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171371 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.171033 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171371 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.171046 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.171371 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.171057 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:22:58.788060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.788026 2578 generic.go:356] "Generic (PLEG): container finished" podID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerID="09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32" exitCode=0 Jul 09 19:22:58.788502 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.788071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerDied","Data":"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32"} Jul 09 19:22:58.788502 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.788108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" event={"ID":"ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700","Type":"ContainerDied","Data":"92264877f5e8461c05e71f3da6c60cf4c86511fc7188c7b934fc6404a86414a8"} Jul 09 19:22:58.788502 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.788129 2578 scope.go:117] "RemoveContainer" containerID="09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32" Jul 09 19:22:58.788502 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.788073 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg" Jul 09 19:22:58.797148 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.797132 2578 scope.go:117] "RemoveContainer" containerID="9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba" Jul 09 19:22:58.804430 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.804410 2578 scope.go:117] "RemoveContainer" containerID="09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32" Jul 09 19:22:58.804685 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:22:58.804666 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32\": container with ID starting with 09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32 not found: ID does not exist" containerID="09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32" Jul 09 19:22:58.804775 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.804696 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32"} err="failed to get container status \"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32\": rpc error: code = NotFound desc = could not find container \"09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32\": container with ID starting with 09024a4d8663ab248e99b2df304b762e7b5bd9df24ffb048e38887cb04094c32 not found: ID does not exist" Jul 09 19:22:58.804866 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.804722 2578 scope.go:117] "RemoveContainer" containerID="9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba" Jul 09 19:22:58.805125 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:22:58.805094 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba\": container with ID starting with 9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba not found: ID does not exist" containerID="9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba" Jul 09 19:22:58.805169 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.805130 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba"} err="failed to get container status \"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba\": rpc error: code = NotFound desc = could not find container \"9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba\": container with ID starting with 9d5fc91d1601dbb26ca0e34f8640df2294b349e3b76605af75667ca2f2da3aba not found: ID does not exist" Jul 09 19:22:58.811724 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.811700 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:22:58.817224 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:58.817205 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bb59c5dc-openshift-default-74f568f6dc-7wnsg"] Jul 09 19:22:59.106378 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:22:59.106295 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" path="/var/lib/kubelet/pods/ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700/volumes" Jul 09 19:23:11.628602 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:11.628563 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:23:45.359304 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:45.359275 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:23:45.362691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:45.362671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:23:48.436877 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.436849 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:23:48.437270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437180 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:23:48.437270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437192 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:23:48.437270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437203 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:23:48.437270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437209 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:23:48.437270 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437271 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="838381c3-a6ed-4d13-86d3-a6cc69256749" containerName="istio-proxy" Jul 09 19:23:48.437433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437279 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:23:48.437433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.437288 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:23:48.440114 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.440097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.442858 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.442837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c6e06b-openshift-default-dockercfg-dr25z\"" Jul 09 19:23:48.451556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.451529 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:23:48.504629 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.504629 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.504888 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.504888 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.504888 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.504888 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.505055 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.505055 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.505055 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.504980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606157 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606360 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606360 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606360 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606360 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606360 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.606691 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.607059 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.607059 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.607059 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.606998 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.608827 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.608802 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.608936 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.608892 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.614447 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.614419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.614534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.614511 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token\") pod \"e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.750838 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.750751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:48.894317 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.894289 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:23:48.896527 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:23:48.896489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1352fc_4fee_409a_854c_dba62bd18e76.slice/crio-a014b5c9cdc516c287a1d556ce5c01e48ddbd4524b22c08be3213ff4224f14e4 WatchSource:0}: Error finding container a014b5c9cdc516c287a1d556ce5c01e48ddbd4524b22c08be3213ff4224f14e4: Status 404 returned error can't find the container with id a014b5c9cdc516c287a1d556ce5c01e48ddbd4524b22c08be3213ff4224f14e4 Jul 09 19:23:48.898716 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.898684 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:23:48.898824 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.898766 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:23:48.898824 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:48.898796 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:23:49.003620 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:49.003525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerStarted","Data":"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5"} Jul 09 19:23:49.003620 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:49.003576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerStarted","Data":"a014b5c9cdc516c287a1d556ce5c01e48ddbd4524b22c08be3213ff4224f14e4"} Jul 09 19:23:49.024937 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:49.024891 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podStartSLOduration=1.024874887 podStartE2EDuration="1.024874887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:49.023756738 +0000 UTC m=+1804.440471149" watchObservedRunningTime="2026-07-09 19:23:49.024874887 +0000 UTC m=+1804.441589296" Jul 09 19:23:49.751433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:49.751397 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:49.756299 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:49.756276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:50.008303 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:50.008216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:50.009235 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:50.009210 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:23:55.731388 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:23:55.731351 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:24:01.931894 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.931858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:24:01.932373 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.932300 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:24:01.932373 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.932315 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf1e6ff-3dd5-4b7b-bbd9-615d7d3ee700" containerName="istio-proxy" Jul 09 19:24:01.937152 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.937130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:01.940138 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.940111 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:24:01.940246 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.940116 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eb3d8a-serving-cert\"" Jul 09 19:24:01.941437 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.941418 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktwt4\"" Jul 09 19:24:01.941531 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.941425 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:24:01.946439 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:01.946413 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:24:02.029176 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.029132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9sf\" (UniqueName: \"kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.029387 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.029261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.029387 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.029350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.130345 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.130312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.130504 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.130385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.130504 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.130414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9sf\" (UniqueName: \"kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.130595 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:24:02.130531 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-serving-cert: secret "maas-api-e2e-inf-b-eb3d8a-serving-cert" not found Jul 09 19:24:02.130643 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:24:02.130630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls podName:2e688535-bbb2-4409-b7bf-3c8878ffb46c nodeName:}" failed. No retries permitted until 2026-07-09 19:24:02.630605984 +0000 UTC m=+1818.047320595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls") pod "maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" (UID: "2e688535-bbb2-4409-b7bf-3c8878ffb46c") : secret "maas-api-e2e-inf-b-eb3d8a-serving-cert" not found Jul 09 19:24:02.131105 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.131086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.139509 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.139470 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9sf\" (UniqueName: \"kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.636596 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.636553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.639083 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.639058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") pod \"maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.849379 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.849338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:02.987125 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.987006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:24:02.990066 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:24:02.990035 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e688535_bbb2_4409_b7bf_3c8878ffb46c.slice/crio-3ace2cf3254a04fab81c1bea7463d53beffb920a255b157cb6727afe990b94fa WatchSource:0}: Error finding container 3ace2cf3254a04fab81c1bea7463d53beffb920a255b157cb6727afe990b94fa: Status 404 returned error can't find the container with id 3ace2cf3254a04fab81c1bea7463d53beffb920a255b157cb6727afe990b94fa Jul 09 19:24:02.991964 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:02.991944 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:24:03.073993 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:03.073956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" event={"ID":"2e688535-bbb2-4409-b7bf-3c8878ffb46c","Type":"ContainerStarted","Data":"3ace2cf3254a04fab81c1bea7463d53beffb920a255b157cb6727afe990b94fa"} Jul 09 19:24:04.081528 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.081487 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerID="9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5" exitCode=0 Jul 09 19:24:04.082051 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.081563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerDied","Data":"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5"} Jul 09 19:24:04.082186 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.082169 2578 scope.go:117] "RemoveContainer" containerID="9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5" Jul 09 19:24:04.083491 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.083451 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:24:04.083591 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.083533 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:24:04.083648 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:04.083588 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:24:05.008315 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.008276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:24:05.088255 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.088219 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerStarted","Data":"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b"} Jul 09 19:24:05.090023 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.089997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" event={"ID":"2e688535-bbb2-4409-b7bf-3c8878ffb46c","Type":"ContainerStarted","Data":"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378"} Jul 09 19:24:05.090166 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.090135 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:05.126383 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.126273 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" podStartSLOduration=2.251480737 podStartE2EDuration="4.126254527s" totalImagesPullingTime="1.87477379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:24:05.125721178 +0000 UTC m=+1820.542435599" watchObservedRunningTime="2026-07-09 19:24:05.126254527 +0000 UTC m=+1820.542968943" Jul 09 19:24:05.752877 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.752836 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:24:05.753101 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.753009 2578 patch_prober.go:28] interesting pod/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 09 19:24:05.753101 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.753056 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 09 19:24:05.929905 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:05.929873 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:24:06.751653 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:06.751611 2578 patch_prober.go:28] interesting pod/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 09 19:24:06.752151 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:06.751676 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 09 19:24:07.752207 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:07.752157 2578 patch_prober.go:28] interesting pod/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 09 19:24:07.752675 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:07.752232 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 09 19:24:08.751555 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:08.751512 2578 patch_prober.go:28] interesting pod/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 09 19:24:08.751790 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:08.751602 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 09 19:24:09.755896 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:09.755865 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:24:09.756337 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:09.756135 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:24:09.756840 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:09.756818 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:24:11.106230 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:11.106206 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:24:53.428523 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:24:53.428479 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:25:04.731660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:25:04.731627 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:25:15.436304 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:25:15.436071 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:25:36.578649 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:25:36.578614 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:26:14.391309 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:14.391266 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:26:14.391999 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:14.391515 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" containerID="cri-o://b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b" gracePeriod=30 Jul 09 19:26:15.693481 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:15.693448 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:26:15.693906 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:15.693699 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" podUID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" containerName="maas-api" containerID="cri-o://3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378" gracePeriod=30 Jul 09 19:26:15.942432 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:15.942408 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:26:16.008500 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.008419 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") pod \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " Jul 09 19:26:16.008657 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.008507 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx9sf\" (UniqueName: \"kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf\") pod \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " Jul 09 19:26:16.008657 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.008545 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca\") pod \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\" (UID: \"2e688535-bbb2-4409-b7bf-3c8878ffb46c\") " Jul 09 19:26:16.008947 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.008922 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e688535-bbb2-4409-b7bf-3c8878ffb46c" (UID: "2e688535-bbb2-4409-b7bf-3c8878ffb46c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:26:16.010551 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.010529 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e688535-bbb2-4409-b7bf-3c8878ffb46c" (UID: "2e688535-bbb2-4409-b7bf-3c8878ffb46c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:26:16.010633 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.010575 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf" (OuterVolumeSpecName: "kube-api-access-fx9sf") pod "2e688535-bbb2-4409-b7bf-3c8878ffb46c" (UID: "2e688535-bbb2-4409-b7bf-3c8878ffb46c"). InnerVolumeSpecName "kube-api-access-fx9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:16.109298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.109271 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e688535-bbb2-4409-b7bf-3c8878ffb46c-openshift-service-ca\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:16.109298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.109294 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e688535-bbb2-4409-b7bf-3c8878ffb46c-maas-api-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:16.109298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.109303 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx9sf\" (UniqueName: \"kubernetes.io/projected/2e688535-bbb2-4409-b7bf-3c8878ffb46c-kube-api-access-fx9sf\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:16.675202 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.675162 2578 generic.go:356] "Generic (PLEG): container finished" podID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" containerID="3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378" exitCode=0 Jul 09 19:26:16.675409 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.675210 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" Jul 09 19:26:16.675409 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.675253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" event={"ID":"2e688535-bbb2-4409-b7bf-3c8878ffb46c","Type":"ContainerDied","Data":"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378"} Jul 09 19:26:16.675409 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.675313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6" event={"ID":"2e688535-bbb2-4409-b7bf-3c8878ffb46c","Type":"ContainerDied","Data":"3ace2cf3254a04fab81c1bea7463d53beffb920a255b157cb6727afe990b94fa"} Jul 09 19:26:16.675409 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.675343 2578 scope.go:117] "RemoveContainer" containerID="3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378" Jul 09 19:26:16.684927 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.684911 2578 scope.go:117] "RemoveContainer" containerID="3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378" Jul 09 19:26:16.685172 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:16.685154 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378\": container with ID starting with 3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378 not found: ID does not exist" containerID="3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378" Jul 09 19:26:16.685218 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.685181 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378"} err="failed to get container status \"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378\": rpc error: code = NotFound desc = could not find container \"3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378\": container with ID starting with 3f47a4ec7912e8ab141b6a24d3b30fc45a2a109a985ffa65a094305bdacc0378 not found: ID does not exist" Jul 09 19:26:16.697153 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.697128 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:26:16.702705 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:16.702683 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eb3d8a-7fdcfdbb69-w5ms6"] Jul 09 19:26:17.106260 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:17.106183 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" path="/var/lib/kubelet/pods/2e688535-bbb2-4409-b7bf-3c8878ffb46c/volumes" Jul 09 19:26:18.473670 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:18.473623 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:26:19.643179 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.643155 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:26:19.692340 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.692247 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerID="b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b" exitCode=0 Jul 09 19:26:19.692340 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.692298 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" Jul 09 19:26:19.692340 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.692320 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerDied","Data":"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b"} Jul 09 19:26:19.692578 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.692377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99" event={"ID":"2d1352fc-4fee-409a-854c-dba62bd18e76","Type":"ContainerDied","Data":"a014b5c9cdc516c287a1d556ce5c01e48ddbd4524b22c08be3213ff4224f14e4"} Jul 09 19:26:19.692578 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.692410 2578 scope.go:117] "RemoveContainer" containerID="b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b" Jul 09 19:26:19.700617 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.700597 2578 scope.go:117] "RemoveContainer" containerID="9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5" Jul 09 19:26:19.708141 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.708122 2578 scope.go:117] "RemoveContainer" containerID="b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b" Jul 09 19:26:19.708423 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:19.708402 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b\": container with ID starting with b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b not found: ID does not exist" containerID="b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b" Jul 09 19:26:19.708476 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.708431 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b"} err="failed to get container status \"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b\": rpc error: code = NotFound desc = could not find container \"b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b\": container with ID starting with b86609db09f8fc39e3e7e3b6378acdaf91a61c8345790e27fda41315efed211b not found: ID does not exist" Jul 09 19:26:19.708476 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.708449 2578 scope.go:117] "RemoveContainer" containerID="9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5" Jul 09 19:26:19.708676 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:19.708658 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5\": container with ID starting with 9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5 not found: ID does not exist" containerID="9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5" Jul 09 19:26:19.708758 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.708685 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5"} err="failed to get container status \"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5\": rpc error: code = NotFound desc = could not find container \"9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5\": container with ID starting with 9692be1edd43c0b53be870e8741445c2407e35510774400acd90edd5e46e69d5 not found: ID does not exist" Jul 09 19:26:19.741285 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741263 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741399 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741306 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741399 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741336 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741399 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741365 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741426 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741517 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741526 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:19.741771 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741569 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741771 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo\") pod \"2d1352fc-4fee-409a-854c-dba62bd18e76\" (UID: \"2d1352fc-4fee-409a-854c-dba62bd18e76\") " Jul 09 19:26:19.741886 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741777 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:19.741945 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741923 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:26:19.742019 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.741999 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.742086 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.742025 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.742155 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.742129 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:19.742678 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.742653 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:19.743675 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.743650 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm" (OuterVolumeSpecName: "kube-api-access-rrjnm") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "kube-api-access-rrjnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:19.743803 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.743653 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:19.743960 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.743938 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:19.744069 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.744045 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d1352fc-4fee-409a-854c-dba62bd18e76" (UID: "2d1352fc-4fee-409a-854c-dba62bd18e76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:26:19.842619 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842590 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842619 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842616 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrjnm\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-kube-api-access-rrjnm\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842619 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842627 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842857 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842635 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842857 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842643 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842857 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842651 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1352fc-4fee-409a-854c-dba62bd18e76-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:19.842857 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:19.842659 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1352fc-4fee-409a-854c-dba62bd18e76-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:20.015693 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:20.015658 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:26:20.019525 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:20.019495 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c6e06b-openshift-default-95c58796f-6ps99"] Jul 09 19:26:21.105364 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:21.105332 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" path="/var/lib/kubelet/pods/2d1352fc-4fee-409a-854c-dba62bd18e76/volumes" Jul 09 19:26:22.524542 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524498 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524886 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524898 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524922 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" containerName="maas-api" Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524929 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" containerName="maas-api" Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524979 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:22.525013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.524987 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e688535-bbb2-4409-b7bf-3c8878ffb46c" containerName="maas-api" Jul 09 19:26:22.529557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.529535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.532419 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.532401 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-08112a-openshift-default-dockercfg-26nkx\"" Jul 09 19:26:22.541444 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.541424 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:22.567794 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fs62\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.567930 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.567930 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.567930 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.567930 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.567930 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.568208 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.567958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.568208 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.568009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.568208 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.568050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.668967 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.668929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.668980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fs62\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669158 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669472 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669472 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669472 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669629 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669686 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.669767 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.669703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.670425 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.670397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.672204 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.672168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.672204 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.672201 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.681171 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.681140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.681445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.681426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fs62\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62\") pod \"e2e-api-a-08112a-openshift-default-797bf7554f-flcmp\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.839533 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.839436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:22.970234 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.970095 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:22.972647 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:26:22.972622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bac284_e198_4081_b452_dfc338d9073e.slice/crio-35dd99357ae0cffd11182467c373dd76ddc4fd6f0af8fc84c27bd870e26dc22d WatchSource:0}: Error finding container 35dd99357ae0cffd11182467c373dd76ddc4fd6f0af8fc84c27bd870e26dc22d: Status 404 returned error can't find the container with id 35dd99357ae0cffd11182467c373dd76ddc4fd6f0af8fc84c27bd870e26dc22d Jul 09 19:26:22.974910 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.974876 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:22.975015 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.974956 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:22.975015 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:22.975003 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:23.712585 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:23.712545 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerStarted","Data":"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a"} Jul 09 19:26:23.712997 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:23.712639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerStarted","Data":"35dd99357ae0cffd11182467c373dd76ddc4fd6f0af8fc84c27bd870e26dc22d"} Jul 09 19:26:23.735185 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:23.735129 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podStartSLOduration=1.735114209 podStartE2EDuration="1.735114209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:26:23.732879178 +0000 UTC m=+1959.149593610" watchObservedRunningTime="2026-07-09 19:26:23.735114209 +0000 UTC m=+1959.151828619" Jul 09 19:26:23.840458 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:23.840421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:24.844571 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:24.844535 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:25.724040 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.724005 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:25.725293 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.725272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:25.730998 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.730971 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:25.731321 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.731306 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:25.731321 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.731321 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:25.731447 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.731376 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1352fc-4fee-409a-854c-dba62bd18e76" containerName="istio-proxy" Jul 09 19:26:25.734093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.734074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.737250 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.737231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:26:25.737370 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.737247 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:26:25.737370 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.737258 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-08112a-serving-cert\"" Jul 09 19:26:25.737370 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.737287 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktwt4\"" Jul 09 19:26:25.744819 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.744794 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:25.799428 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.799395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.799620 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.799457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.799686 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.799611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.900902 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.900868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.901303 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.900948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.901303 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:25.901008 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-serving-cert: secret "maas-api-e2e-api-a-08112a-serving-cert" not found Jul 09 19:26:25.901303 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.901024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.901303 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:25.901065 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls podName:e4b4b414-7684-4d70-bf33-7b700870129b nodeName:}" failed. No retries permitted until 2026-07-09 19:26:26.401048165 +0000 UTC m=+1961.817762553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls") pod "maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" (UID: "e4b4b414-7684-4d70-bf33-7b700870129b") : secret "maas-api-e2e-api-a-08112a-serving-cert" not found Jul 09 19:26:25.901599 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.901581 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:25.911314 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:25.911287 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:26.405924 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:26.405883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:26.408469 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:26.408443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") pod \"maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:26.645865 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:26.645827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:26.775877 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:26.775853 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:26.777581 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:26:26.777557 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b4b414_7684_4d70_bf33_7b700870129b.slice/crio-014a9b7fe0bdafe05278044fde32c9757a37cefc0d59342e59a07f888bfe61d7 WatchSource:0}: Error finding container 014a9b7fe0bdafe05278044fde32c9757a37cefc0d59342e59a07f888bfe61d7: Status 404 returned error can't find the container with id 014a9b7fe0bdafe05278044fde32c9757a37cefc0d59342e59a07f888bfe61d7 Jul 09 19:26:27.734874 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:27.734837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" event={"ID":"e4b4b414-7684-4d70-bf33-7b700870129b","Type":"ContainerStarted","Data":"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48"} Jul 09 19:26:27.735348 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:27.734927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" event={"ID":"e4b4b414-7684-4d70-bf33-7b700870129b","Type":"ContainerStarted","Data":"014a9b7fe0bdafe05278044fde32c9757a37cefc0d59342e59a07f888bfe61d7"} Jul 09 19:26:27.735348 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:27.735047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:27.752346 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:27.752290 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" podStartSLOduration=2.243974172 podStartE2EDuration="2.752275205s" totalImagesPullingTime="508.301033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:26:27.751620167 +0000 UTC m=+1963.168334578" watchObservedRunningTime="2026-07-09 19:26:27.752275205 +0000 UTC m=+1963.168989668" Jul 09 19:26:33.737632 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:33.737599 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:26:33.747830 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:33.747793 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:41.764956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:41.764872 2578 patch_prober.go:28] interesting pod/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 19:26:41.765335 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:41.764947 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 19:26:42.808367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.808332 2578 generic.go:356] "Generic (PLEG): container finished" podID="10bac284-e198-4081-b452-dfc338d9073e" containerID="280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a" exitCode=0 Jul 09 19:26:42.808856 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.808382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerDied","Data":"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a"} Jul 09 19:26:42.808856 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.808765 2578 scope.go:117] "RemoveContainer" containerID="280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a" Jul 09 19:26:42.809751 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.809694 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:42.809872 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.809807 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:42.809872 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:42.809862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:26:43.814739 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:43.814702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerStarted","Data":"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e"} Jul 09 19:26:43.839830 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:43.839806 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:43.839988 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:43.839969 2578 patch_prober.go:28] interesting pod/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 19:26:43.840031 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:43.840012 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 19:26:44.840465 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:44.840432 2578 patch_prober.go:28] interesting pod/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 19:26:44.840887 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:44.840484 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 19:26:45.840367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:45.840330 2578 patch_prober.go:28] interesting pod/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 19:26:45.840531 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:45.840388 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 19:26:46.840802 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:46.840768 2578 patch_prober.go:28] interesting pod/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 09 19:26:46.841169 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:46.840820 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 09 19:26:47.221071 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.221040 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:47.221346 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.221284 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" podUID="e4b4b414-7684-4d70-bf33-7b700870129b" containerName="maas-api" containerID="cri-o://d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48" gracePeriod=30 Jul 09 19:26:47.516607 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.516582 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:47.705018 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.704979 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca\") pod \"e4b4b414-7684-4d70-bf33-7b700870129b\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " Jul 09 19:26:47.705186 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.705148 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj\") pod \"e4b4b414-7684-4d70-bf33-7b700870129b\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " Jul 09 19:26:47.705228 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.705212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") pod \"e4b4b414-7684-4d70-bf33-7b700870129b\" (UID: \"e4b4b414-7684-4d70-bf33-7b700870129b\") " Jul 09 19:26:47.705367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.705342 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4b4b414-7684-4d70-bf33-7b700870129b" (UID: "e4b4b414-7684-4d70-bf33-7b700870129b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:26:47.705539 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.705520 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4b4b414-7684-4d70-bf33-7b700870129b-openshift-service-ca\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:47.707210 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.707186 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4b4b414-7684-4d70-bf33-7b700870129b" (UID: "e4b4b414-7684-4d70-bf33-7b700870129b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:26:47.707285 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.707227 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj" (OuterVolumeSpecName: "kube-api-access-jkpbj") pod "e4b4b414-7684-4d70-bf33-7b700870129b" (UID: "e4b4b414-7684-4d70-bf33-7b700870129b"). InnerVolumeSpecName "kube-api-access-jkpbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:47.806214 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.806115 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/e4b4b414-7684-4d70-bf33-7b700870129b-kube-api-access-jkpbj\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:47.806214 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.806152 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4b4b414-7684-4d70-bf33-7b700870129b-maas-api-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:47.834685 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.834647 2578 generic.go:356] "Generic (PLEG): container finished" podID="e4b4b414-7684-4d70-bf33-7b700870129b" containerID="d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48" exitCode=0 Jul 09 19:26:47.834898 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.834697 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" Jul 09 19:26:47.834898 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.834752 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" event={"ID":"e4b4b414-7684-4d70-bf33-7b700870129b","Type":"ContainerDied","Data":"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48"} Jul 09 19:26:47.834898 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.834807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk" event={"ID":"e4b4b414-7684-4d70-bf33-7b700870129b","Type":"ContainerDied","Data":"014a9b7fe0bdafe05278044fde32c9757a37cefc0d59342e59a07f888bfe61d7"} Jul 09 19:26:47.834898 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.834831 2578 scope.go:117] "RemoveContainer" containerID="d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48" Jul 09 19:26:47.843814 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.843757 2578 scope.go:117] "RemoveContainer" containerID="d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48" Jul 09 19:26:47.844140 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:47.844024 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48\": container with ID starting with d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48 not found: ID does not exist" containerID="d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48" Jul 09 19:26:47.844140 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.844051 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48"} err="failed to get container status \"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48\": rpc error: code = NotFound desc = could not find container \"d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48\": container with ID starting with d4d062a57207b8738c5bf5b8321e5b5f42f925d3b774fcd8c9e64cf1ff46fc48 not found: ID does not exist" Jul 09 19:26:47.849487 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.849464 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:47.849847 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.849818 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:47.851136 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.851118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:47.855772 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.855749 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:47.860011 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:47.859994 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08112a-5f5f9d955f-c6krk"] Jul 09 19:26:48.328644 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:48.328606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:26:49.106254 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:49.106225 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b4b414-7684-4d70-bf33-7b700870129b" path="/var/lib/kubelet/pods/e4b4b414-7684-4d70-bf33-7b700870129b/volumes" Jul 09 19:26:53.053765 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:53.053697 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:53.054300 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:53.053947 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" containerID="cri-o://cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e" gracePeriod=30 Jul 09 19:26:58.301024 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.300999 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:58.402584 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402562 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402745 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402745 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402642 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402745 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fs62\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402745 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402700 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402756 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402794 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402825 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.402956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402850 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket\") pod \"10bac284-e198-4081-b452-dfc338d9073e\" (UID: \"10bac284-e198-4081-b452-dfc338d9073e\") " Jul 09 19:26:58.403176 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402968 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:26:58.403176 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.402979 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:58.403176 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403076 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:58.403314 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403208 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:58.403314 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403235 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.403314 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403254 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10bac284-e198-4081-b452-dfc338d9073e-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.403314 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403267 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.403793 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.403771 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data" (OuterVolumeSpecName: "istio-data") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:58.405400 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.405369 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token" (OuterVolumeSpecName: "istio-token") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:58.405400 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.405371 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:26:58.405534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.405449 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:26:58.405534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.405494 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62" (OuterVolumeSpecName: "kube-api-access-7fs62") pod "10bac284-e198-4081-b452-dfc338d9073e" (UID: "10bac284-e198-4081-b452-dfc338d9073e"). InnerVolumeSpecName "kube-api-access-7fs62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:26:58.504323 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504288 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10bac284-e198-4081-b452-dfc338d9073e-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.504323 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504316 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fs62\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-kube-api-access-7fs62\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.504323 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504330 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10bac284-e198-4081-b452-dfc338d9073e-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.504534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504342 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.504534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504353 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.504534 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.504365 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10bac284-e198-4081-b452-dfc338d9073e-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:26:58.886585 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.886554 2578 generic.go:356] "Generic (PLEG): container finished" podID="10bac284-e198-4081-b452-dfc338d9073e" containerID="cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e" exitCode=0 Jul 09 19:26:58.886748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.886598 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" Jul 09 19:26:58.886748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.886637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerDied","Data":"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e"} Jul 09 19:26:58.886748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.886689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp" event={"ID":"10bac284-e198-4081-b452-dfc338d9073e","Type":"ContainerDied","Data":"35dd99357ae0cffd11182467c373dd76ddc4fd6f0af8fc84c27bd870e26dc22d"} Jul 09 19:26:58.886748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.886710 2578 scope.go:117] "RemoveContainer" containerID="cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e" Jul 09 19:26:58.895708 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.895687 2578 scope.go:117] "RemoveContainer" containerID="280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a" Jul 09 19:26:58.902969 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.902950 2578 scope.go:117] "RemoveContainer" containerID="cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e" Jul 09 19:26:58.903234 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:58.903215 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e\": container with ID starting with cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e not found: ID does not exist" containerID="cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e" Jul 09 19:26:58.903283 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.903243 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e"} err="failed to get container status \"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e\": rpc error: code = NotFound desc = could not find container \"cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e\": container with ID starting with cb563dd72fd28d5bdcea4cf2ae82ef121e4843676eb24fa913fdeb20e7887f1e not found: ID does not exist" Jul 09 19:26:58.903283 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.903262 2578 scope.go:117] "RemoveContainer" containerID="280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a" Jul 09 19:26:58.903484 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:26:58.903467 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a\": container with ID starting with 280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a not found: ID does not exist" containerID="280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a" Jul 09 19:26:58.903573 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.903495 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a"} err="failed to get container status \"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a\": rpc error: code = NotFound desc = could not find container \"280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a\": container with ID starting with 280e4f0c23ed331e7201417b6d338d81b9e4fff40a716df257af77fa3d3ac05a not found: ID does not exist" Jul 09 19:26:58.909309 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.909288 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:58.913140 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:58.913118 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-08112a-openshift-default-797bf7554f-flcmp"] Jul 09 19:26:59.106337 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:26:59.106303 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10bac284-e198-4081-b452-dfc338d9073e" path="/var/lib/kubelet/pods/10bac284-e198-4081-b452-dfc338d9073e/volumes" Jul 09 19:27:01.405923 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.405888 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:27:01.406282 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406241 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b4b414-7684-4d70-bf33-7b700870129b" containerName="maas-api" Jul 09 19:27:01.406282 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406254 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b4b414-7684-4d70-bf33-7b700870129b" containerName="maas-api" Jul 09 19:27:01.406282 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406273 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.406282 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406278 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.406407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406290 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.406407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406300 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.406407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406348 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b4b414-7684-4d70-bf33-7b700870129b" containerName="maas-api" Jul 09 19:27:01.406407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406359 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.406407 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.406367 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bac284-e198-4081-b452-dfc338d9073e" containerName="istio-proxy" Jul 09 19:27:01.409363 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.409344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.411956 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.411935 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5d8f8f-openshift-default-dockercfg-bx4rr\"" Jul 09 19:27:01.421480 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.421455 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:27:01.429405 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429405 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429557 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph5f\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429667 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429667 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.429748 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.429721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531104 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531104 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531445 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531576 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531576 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531461 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531576 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph5f\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531576 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531518 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531813 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531650 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.531860 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.531847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.532195 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.532174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.533588 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.533568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.533689 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.533675 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.540145 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.540119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.540561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.540540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph5f\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f\") pod \"e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.719426 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.719336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:01.852814 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.852789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:27:01.854549 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:27:01.854521 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1360583_9297_4a41_a03a_4ccd391d0c6b.slice/crio-bb94eb96f179a69fcf380e22dfbccea1cbe5e78859d9a857fa57badc3339d5b5 WatchSource:0}: Error finding container bb94eb96f179a69fcf380e22dfbccea1cbe5e78859d9a857fa57badc3339d5b5: Status 404 returned error can't find the container with id bb94eb96f179a69fcf380e22dfbccea1cbe5e78859d9a857fa57badc3339d5b5 Jul 09 19:27:01.856676 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.856637 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:01.856798 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.856717 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:01.856798 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.856785 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:01.902868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:01.902839 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerStarted","Data":"bb94eb96f179a69fcf380e22dfbccea1cbe5e78859d9a857fa57badc3339d5b5"} Jul 09 19:27:02.908567 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:02.908533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerStarted","Data":"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee"} Jul 09 19:27:02.930441 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:02.930396 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podStartSLOduration=1.9303825159999999 podStartE2EDuration="1.930382516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:02.929309399 +0000 UTC m=+1998.346023812" watchObservedRunningTime="2026-07-09 19:27:02.930382516 +0000 UTC m=+1998.347096921" Jul 09 19:27:03.720237 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:03.720201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:03.725194 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:03.725171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:03.913578 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:03.913547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:03.914566 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:03.914548 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:04.719928 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.719892 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:27:04.922277 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.922248 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:27:04.925679 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.925658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:04.928660 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.928640 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktwt4\"" Jul 09 19:27:04.928810 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.928692 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:27:04.929949 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.929931 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:27:04.930176 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.929948 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5d8f8f-serving-cert\"" Jul 09 19:27:04.936756 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.936702 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:27:04.962937 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.962915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:04.963081 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.962987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:04.963237 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:04.963209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb95p\" (UniqueName: \"kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.064504 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.064408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb95p\" (UniqueName: \"kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.064504 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.064458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.064696 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.064506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.064696 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:27:05.064617 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-serving-cert: secret "maas-api-e2e-shared-a-5d8f8f-serving-cert" not found Jul 09 19:27:05.064696 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:27:05.064667 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls podName:0f497d01-b6d9-4ffe-b3ab-1bc187433e7c nodeName:}" failed. No retries permitted until 2026-07-09 19:27:05.564651904 +0000 UTC m=+2000.981366293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls") pod "maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" (UID: "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c") : secret "maas-api-e2e-shared-a-5d8f8f-serving-cert" not found Jul 09 19:27:05.065222 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.065199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.073060 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.073036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb95p\" (UniqueName: \"kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.569244 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:05.569204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:05.569429 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:27:05.569360 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-serving-cert: secret "maas-api-e2e-shared-a-5d8f8f-serving-cert" not found Jul 09 19:27:05.569429 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:27:05.569423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls podName:0f497d01-b6d9-4ffe-b3ab-1bc187433e7c nodeName:}" failed. No retries permitted until 2026-07-09 19:27:06.569407378 +0000 UTC m=+2001.986121766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls") pod "maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" (UID: "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c") : secret "maas-api-e2e-shared-a-5d8f8f-serving-cert" not found Jul 09 19:27:06.577629 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:06.577589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:06.580134 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:06.580105 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:06.738034 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:06.737995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:06.868438 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:06.868356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:27:06.870666 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:27:06.870638 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f497d01_b6d9_4ffe_b3ab_1bc187433e7c.slice/crio-2ddeac62a960b9f0b9799f7ab2f7522ae53dc48418ed8736df5b8aa7eb6b926d WatchSource:0}: Error finding container 2ddeac62a960b9f0b9799f7ab2f7522ae53dc48418ed8736df5b8aa7eb6b926d: Status 404 returned error can't find the container with id 2ddeac62a960b9f0b9799f7ab2f7522ae53dc48418ed8736df5b8aa7eb6b926d Jul 09 19:27:06.929198 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:06.929152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" event={"ID":"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c","Type":"ContainerStarted","Data":"2ddeac62a960b9f0b9799f7ab2f7522ae53dc48418ed8736df5b8aa7eb6b926d"} Jul 09 19:27:07.935470 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:07.935434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" event={"ID":"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c","Type":"ContainerStarted","Data":"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e"} Jul 09 19:27:07.935901 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:07.935567 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:07.955242 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:07.955195 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" podStartSLOduration=3.620570181 podStartE2EDuration="3.95517927s" totalImagesPullingTime="334.609089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:07.952917419 +0000 UTC m=+2003.369631840" watchObservedRunningTime="2026-07-09 19:27:07.95517927 +0000 UTC m=+2003.371893680" Jul 09 19:27:13.945383 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:13.945342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:27:16.937533 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:16.937494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:27:19.112127 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.111253 2578 patch_prober.go:28] interesting pod/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:44594->10.134.0.35:15021: read: connection reset by peer" start-of-body= Jul 09 19:27:19.112127 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.111311 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:44594->10.134.0.35:15021: read: connection reset by peer" Jul 09 19:27:19.995968 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.995927 2578 generic.go:356] "Generic (PLEG): container finished" podID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerID="976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee" exitCode=0 Jul 09 19:27:19.996165 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.996001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerDied","Data":"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee"} Jul 09 19:27:19.996419 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.996403 2578 scope.go:117] "RemoveContainer" containerID="976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee" Jul 09 19:27:19.997334 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.997307 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:19.997421 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.997372 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:19.997421 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:19.997403 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 09 19:27:21.003284 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:21.003253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerStarted","Data":"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3"} Jul 09 19:27:21.720496 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:21.720460 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:21.720679 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:21.720662 2578 patch_prober.go:28] interesting pod/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 09 19:27:21.720776 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:21.720714 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 09 19:27:22.720202 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:22.720162 2578 patch_prober.go:28] interesting pod/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 09 19:27:22.720600 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:22.720222 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 09 19:27:23.720378 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:23.720342 2578 patch_prober.go:28] interesting pod/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 09 19:27:23.720826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:23.720402 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 09 19:27:24.723654 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:24.723624 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:24.724260 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:24.724008 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:27:24.724778 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:27:24.724746 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:28:00.428625 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:00.428592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:28:24.030405 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:24.030323 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:28:39.631964 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:39.631928 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:28:45.391919 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:45.391888 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:28:45.396389 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:45.396366 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:28:58.136823 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:28:58.136789 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:29:16.822097 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:29:16.822057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:29:32.425591 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:29:32.425561 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:29:52.231305 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:29:52.231220 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:30:11.527567 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:11.527529 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:30:30.127665 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:30.127619 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:30:36.248478 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.248443 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:30:36.251928 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.251913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.254500 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.254480 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"openshift-service-ca.crt\"" Jul 09 19:30:36.255844 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.255821 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"default-dockercfg-crx4n\"" Jul 09 19:30:36.255957 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.255848 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"sub-test-model-5d8f8f-kserve-self-signed-certs\"" Jul 09 19:30:36.255957 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.255861 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d8f8f\"/\"kube-root-ca.crt\"" Jul 09 19:30:36.260998 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.260978 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:30:36.328948 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.328914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vzh\" (UniqueName: \"kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.328948 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.328952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.329164 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.328970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.329164 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.329013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.329235 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.329155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.329235 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.329188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.429813 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.429813 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430086 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vzh\" (UniqueName: \"kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430086 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430086 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430086 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.429912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430297 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.430252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430297 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.430258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.430418 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.430402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.432224 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.432203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.432505 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.432485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.437897 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.437876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vzh\" (UniqueName: \"kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh\") pod \"sub-test-model-5d8f8f-kserve-645dd76597-2vswv\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.561354 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.561270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:36.688982 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.688959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:30:36.691158 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:30:36.691125 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06ef4e8_7104_447b_9652_ac7dfce3b799.slice/crio-740c46c10b25970a142185670ffbca789c01ab12f9be5f87fd4b6c7d045e930b WatchSource:0}: Error finding container 740c46c10b25970a142185670ffbca789c01ab12f9be5f87fd4b6c7d045e930b: Status 404 returned error can't find the container with id 740c46c10b25970a142185670ffbca789c01ab12f9be5f87fd4b6c7d045e930b Jul 09 19:30:36.904756 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.904694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" event={"ID":"e06ef4e8-7104-447b-9652-ac7dfce3b799","Type":"ContainerStarted","Data":"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1"} Jul 09 19:30:36.904756 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.904761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" event={"ID":"e06ef4e8-7104-447b-9652-ac7dfce3b799","Type":"ContainerStarted","Data":"740c46c10b25970a142185670ffbca789c01ab12f9be5f87fd4b6c7d045e930b"} Jul 09 19:30:36.921667 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:36.921611 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" podStartSLOduration=0.921592367 podStartE2EDuration="921.592367ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:36.921255478 +0000 UTC m=+2212.337969898" watchObservedRunningTime="2026-07-09 19:30:36.921592367 +0000 UTC m=+2212.338306859" Jul 09 19:30:46.333549 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:46.333504 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:30:46.562225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:46.562186 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:46.562225 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:46.562230 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:46.574712 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:46.574685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:30:46.962701 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:30:46.962668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:31:12.534565 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:31:12.534485 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:31:34.436367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:31:34.436332 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:31:53.528715 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:31:53.528681 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:32:00.338491 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:00.338462 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:32:12.723237 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:12.723199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:32:15.358378 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.358338 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" podUID="e06ef4e8-7104-447b-9652-ac7dfce3b799" containerName="main" containerID="cri-o://b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1" gracePeriod=45 Jul 09 19:32:15.597406 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.597380 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:32:15.699357 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699327 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6vzh\" (UniqueName: \"kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699543 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699543 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699409 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699543 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699438 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699543 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699454 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699543 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm\") pod \"e06ef4e8-7104-447b-9652-ac7dfce3b799\" (UID: \"e06ef4e8-7104-447b-9652-ac7dfce3b799\") " Jul 09 19:32:15.699854 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699698 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:15.699854 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699770 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home" (OuterVolumeSpecName: "home") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:15.699854 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.699791 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache" (OuterVolumeSpecName: "model-cache") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:15.701593 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.701572 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm" (OuterVolumeSpecName: "dshm") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:32:15.701689 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.701634 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:32:15.701756 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.701722 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh" (OuterVolumeSpecName: "kube-api-access-r6vzh") pod "e06ef4e8-7104-447b-9652-ac7dfce3b799" (UID: "e06ef4e8-7104-447b-9652-ac7dfce3b799"). InnerVolumeSpecName "kube-api-access-r6vzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:32:15.800529 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800500 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vzh\" (UniqueName: \"kubernetes.io/projected/e06ef4e8-7104-447b-9652-ac7dfce3b799-kube-api-access-r6vzh\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:15.800529 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800527 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-tmp-dir\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:15.800676 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800538 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-home\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:15.800676 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800546 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-model-cache\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:15.800676 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800555 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06ef4e8-7104-447b-9652-ac7dfce3b799-tls-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:15.800676 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:15.800562 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06ef4e8-7104-447b-9652-ac7dfce3b799-dshm\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:32:16.352817 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.352785 2578 generic.go:356] "Generic (PLEG): container finished" podID="e06ef4e8-7104-447b-9652-ac7dfce3b799" containerID="b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1" exitCode=0 Jul 09 19:32:16.352983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.352830 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" Jul 09 19:32:16.352983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.352860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" event={"ID":"e06ef4e8-7104-447b-9652-ac7dfce3b799","Type":"ContainerDied","Data":"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1"} Jul 09 19:32:16.352983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.352901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv" event={"ID":"e06ef4e8-7104-447b-9652-ac7dfce3b799","Type":"ContainerDied","Data":"740c46c10b25970a142185670ffbca789c01ab12f9be5f87fd4b6c7d045e930b"} Jul 09 19:32:16.352983 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.352920 2578 scope.go:117] "RemoveContainer" containerID="b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1" Jul 09 19:32:16.361792 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.361603 2578 scope.go:117] "RemoveContainer" containerID="b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1" Jul 09 19:32:16.362012 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:32:16.361851 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1\": container with ID starting with b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1 not found: ID does not exist" containerID="b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1" Jul 09 19:32:16.362012 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.361876 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1"} err="failed to get container status \"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1\": rpc error: code = NotFound desc = could not find container \"b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1\": container with ID starting with b51b9c90081b287a9e1ab43ee19882ede02befa208e213ea280c4ad62b32ffb1 not found: ID does not exist" Jul 09 19:32:16.374599 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.374577 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:32:16.378519 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:16.378496 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d8f8f/sub-test-model-5d8f8f-kserve-645dd76597-2vswv"] Jul 09 19:32:17.105791 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:17.105753 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06ef4e8-7104-447b-9652-ac7dfce3b799" path="/var/lib/kubelet/pods/e06ef4e8-7104-447b-9652-ac7dfce3b799/volumes" Jul 09 19:32:25.230791 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:25.230755 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:32:49.448808 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:32:49.448766 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:33:09.437364 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:09.437323 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:33:25.634009 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:25.633972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:33:40.739333 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:40.739301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:33:45.426850 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:45.426821 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:33:45.432749 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:45.432701 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4hc8_0824ce37-5f83-4f67-bf0f-56c7f723f3d2/ovn-acl-logging/0.log" Jul 09 19:33:53.743397 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:53.743365 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:33:53.743827 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:53.743629 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerName="maas-api" containerID="cri-o://bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e" gracePeriod=30 Jul 09 19:33:53.990638 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:53.990610 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:33:54.087438 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.087354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb95p\" (UniqueName: \"kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p\") pod \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " Jul 09 19:33:54.087595 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.087460 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") pod \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " Jul 09 19:33:54.087595 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.087495 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca\") pod \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\" (UID: \"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c\") " Jul 09 19:33:54.087959 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.087924 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" (UID: "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:33:54.089444 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.089420 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" (UID: "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:33:54.089516 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.089448 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p" (OuterVolumeSpecName: "kube-api-access-cb95p") pod "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" (UID: "0f497d01-b6d9-4ffe-b3ab-1bc187433e7c"). InnerVolumeSpecName "kube-api-access-cb95p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:33:54.188997 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.188958 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb95p\" (UniqueName: \"kubernetes.io/projected/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-kube-api-access-cb95p\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:33:54.188997 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.188998 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-maas-api-tls\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:33:54.188997 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.189014 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c-openshift-service-ca\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:33:54.793430 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.793397 2578 generic.go:356] "Generic (PLEG): container finished" podID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerID="bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e" exitCode=0 Jul 09 19:33:54.793887 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.793442 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" Jul 09 19:33:54.793887 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.793450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" event={"ID":"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c","Type":"ContainerDied","Data":"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e"} Jul 09 19:33:54.793887 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.793491 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" event={"ID":"0f497d01-b6d9-4ffe-b3ab-1bc187433e7c","Type":"ContainerDied","Data":"2ddeac62a960b9f0b9799f7ab2f7522ae53dc48418ed8736df5b8aa7eb6b926d"} Jul 09 19:33:54.793887 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.793513 2578 scope.go:117] "RemoveContainer" containerID="bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e" Jul 09 19:33:54.802724 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.802698 2578 scope.go:117] "RemoveContainer" containerID="bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e" Jul 09 19:33:54.803005 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:33:54.802988 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e\": container with ID starting with bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e not found: ID does not exist" containerID="bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e" Jul 09 19:33:54.803048 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.803013 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e"} err="failed to get container status \"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e\": rpc error: code = NotFound desc = could not find container \"bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e\": container with ID starting with bfcca7c8e03fc4810a498af8ad09074e61cd0ad4df85b293ef73ffba45d5c16e not found: ID does not exist" Jul 09 19:33:54.815135 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.815102 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:33:54.819565 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:54.819528 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx"] Jul 09 19:33:55.106468 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:55.106390 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" path="/var/lib/kubelet/pods/0f497d01-b6d9-4ffe-b3ab-1bc187433e7c/volumes" Jul 09 19:33:56.940436 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:56.940382 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d8f8f-75c9bcbf46-xwnfx" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 09 19:33:57.623576 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:33:57.623542 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:34:00.000822 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:00.000788 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:34:00.001256 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:00.000981 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" containerID="cri-o://8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3" gracePeriod=30 Jul 09 19:34:05.257772 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.257716 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:34:05.396073 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396034 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396099 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396138 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396187 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396208 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396276 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396263 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396289 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dph5f\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396333 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket\") pod \"b1360583-9297-4a41-a03a-4ccd391d0c6b\" (UID: \"b1360583-9297-4a41-a03a-4ccd391d0c6b\") " Jul 09 19:34:05.396592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396491 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:05.396592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:05.396592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396579 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:34:05.396826 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396725 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:05.396875 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396821 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1360583-9297-4a41-a03a-4ccd391d0c6b-istiod-ca-cert\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.396875 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396843 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.396875 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.396857 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-workload-certs\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.397528 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.397491 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:05.398383 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.398352 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:34:05.398503 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.398485 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:05.398553 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.398502 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f" (OuterVolumeSpecName: "kube-api-access-dph5f") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "kube-api-access-dph5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:05.398767 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.398706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1360583-9297-4a41-a03a-4ccd391d0c6b" (UID: "b1360583-9297-4a41-a03a-4ccd391d0c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:05.498213 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498179 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-credential-socket\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.498213 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498211 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-token\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.498213 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498221 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-podinfo\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.498453 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498230 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-envoy\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.498453 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498237 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1360583-9297-4a41-a03a-4ccd391d0c6b-istio-data\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.498453 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.498245 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dph5f\" (UniqueName: \"kubernetes.io/projected/b1360583-9297-4a41-a03a-4ccd391d0c6b-kube-api-access-dph5f\") on node \"ip-10-0-131-3.ec2.internal\" DevicePath \"\"" Jul 09 19:34:05.845367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.845282 2578 generic.go:356] "Generic (PLEG): container finished" podID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerID="8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3" exitCode=0 Jul 09 19:34:05.845367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.845331 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" Jul 09 19:34:05.845367 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.845355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerDied","Data":"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3"} Jul 09 19:34:05.845590 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.845400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm" event={"ID":"b1360583-9297-4a41-a03a-4ccd391d0c6b","Type":"ContainerDied","Data":"bb94eb96f179a69fcf380e22dfbccea1cbe5e78859d9a857fa57badc3339d5b5"} Jul 09 19:34:05.845590 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.845421 2578 scope.go:117] "RemoveContainer" containerID="8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3" Jul 09 19:34:05.854652 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.854627 2578 scope.go:117] "RemoveContainer" containerID="976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee" Jul 09 19:34:05.862709 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.862689 2578 scope.go:117] "RemoveContainer" containerID="8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3" Jul 09 19:34:05.863040 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:34:05.863020 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3\": container with ID starting with 8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3 not found: ID does not exist" containerID="8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3" Jul 09 19:34:05.863096 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.863050 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3"} err="failed to get container status \"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3\": rpc error: code = NotFound desc = could not find container \"8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3\": container with ID starting with 8e5bcac415092afdcf95730936b8664b2d26cf060624e9916d02a14a8a2713c3 not found: ID does not exist" Jul 09 19:34:05.863096 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.863071 2578 scope.go:117] "RemoveContainer" containerID="976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee" Jul 09 19:34:05.863349 ip-10-0-131-3 kubenswrapper[2578]: E0709 19:34:05.863324 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee\": container with ID starting with 976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee not found: ID does not exist" containerID="976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee" Jul 09 19:34:05.863439 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.863353 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee"} err="failed to get container status \"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee\": rpc error: code = NotFound desc = could not find container \"976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee\": container with ID starting with 976d8a47f0909428a0e9685026d76f652c956e35bcbb7bd7f3b573fb22cd44ee not found: ID does not exist" Jul 09 19:34:05.871055 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.869574 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:34:05.875433 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:05.875398 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5d8f8f-openshift-default-59d4f6df5f-228fm"] Jul 09 19:34:07.105500 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:07.105423 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" path="/var/lib/kubelet/pods/b1360583-9297-4a41-a03a-4ccd391d0c6b/volumes" Jul 09 19:34:09.831040 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:09.830999 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ddzsn"] Jul 09 19:34:13.457070 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:13.457041 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-gm7r4_f66acc91-f60d-4884-91c5-a8f485d9a1f7/copy-manifests/0.log" Jul 09 19:34:13.474945 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:13.474896 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-gm7r4_f66acc91-f60d-4884-91c5-a8f485d9a1f7/manager/0.log" Jul 09 19:34:16.153750 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:16.153695 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ddzsn_f00f672c-4888-402a-973b-67cc103c8dcf/limitador/0.log" Jul 09 19:34:16.633461 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:16.633423 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689d8rk8d_761dce66-2b21-4b62-8aa7-ed63a5aeda95/istio-proxy/0.log" Jul 09 19:34:16.757373 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:16.757343 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-h2fmt_48c0afb1-990b-4773-a7f4-bfacd07d93d1/discovery/0.log" Jul 09 19:34:17.115658 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:17.115580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/1.log" Jul 09 19:34:18.186577 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:18.186548 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbbw4d2_a0531612-9be8-421b-92c6-321cf7d8a50f/main/0.log" Jul 09 19:34:25.649961 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:25.649933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g2nbg_2c4b166f-98c3-43d4-a8b6-6872182419f6/global-pull-secret-syncer/0.log" Jul 09 19:34:25.884897 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:25.884867 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pc4s2_0d580e6d-4c52-4ed8-b1e3-f2776b7c5bd7/konnectivity-agent/0.log" Jul 09 19:34:25.953654 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:25.953575 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-3.ec2.internal_7acd9c4c33c3ec7d300bea1511a846a1/haproxy/0.log" Jul 09 19:34:32.334804 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:32.334710 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ddzsn_f00f672c-4888-402a-973b-67cc103c8dcf/limitador/0.log" Jul 09 19:34:34.314322 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.314292 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-87bbc467f-pzhnk_b096a242-1d66-46e6-a7ae-48771b467891/metrics-server/0.log" Jul 09 19:34:34.344332 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.344309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-8n882_e28520c9-0718-4e99-a006-98a018a7e90b/monitoring-plugin/0.log" Jul 09 19:34:34.452332 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.452302 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h8zb_37952e3a-9fb9-4860-b3c6-d02429d75b58/node-exporter/0.log" Jul 09 19:34:34.472621 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.472595 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h8zb_37952e3a-9fb9-4860-b3c6-d02429d75b58/kube-rbac-proxy/0.log" Jul 09 19:34:34.493868 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.493845 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h8zb_37952e3a-9fb9-4860-b3c6-d02429d75b58/init-textfile/0.log" Jul 09 19:34:34.606031 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.605956 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-fzw6q_12911dea-c8cc-4a03-8ca7-7bfd2f269a1e/kube-rbac-proxy-main/0.log" Jul 09 19:34:34.628923 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.628894 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-fzw6q_12911dea-c8cc-4a03-8ca7-7bfd2f269a1e/kube-rbac-proxy-self/0.log" Jul 09 19:34:34.650349 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.650325 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-fzw6q_12911dea-c8cc-4a03-8ca7-7bfd2f269a1e/openshift-state-metrics/0.log" Jul 09 19:34:34.686974 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.686946 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/prometheus/0.log" Jul 09 19:34:34.708431 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.708404 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/config-reloader/0.log" Jul 09 19:34:34.738435 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.738408 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/thanos-sidecar/0.log" Jul 09 19:34:34.760485 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.760461 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/kube-rbac-proxy-web/0.log" Jul 09 19:34:34.782114 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.782091 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/kube-rbac-proxy/0.log" Jul 09 19:34:34.804269 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.804242 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/kube-rbac-proxy-thanos/0.log" Jul 09 19:34:34.828641 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:34.828617 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ef756e-496a-4a1a-8fb3-8d4c73cdae6b/init-config-reloader/0.log" Jul 09 19:34:35.012693 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.012665 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/thanos-query/0.log" Jul 09 19:34:35.034093 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.034067 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/kube-rbac-proxy-web/0.log" Jul 09 19:34:35.056607 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.056580 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/kube-rbac-proxy/0.log" Jul 09 19:34:35.079645 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.079621 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/prom-label-proxy/0.log" Jul 09 19:34:35.102718 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.102692 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/kube-rbac-proxy-rules/0.log" Jul 09 19:34:35.123656 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.123631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-799b685df8-sxvgk_fa81b2e5-bab7-468b-b239-414689dad316/kube-rbac-proxy-metrics/0.log" Jul 09 19:34:35.415589 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415556 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf"] Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415895 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415907 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415917 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerName="maas-api" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415925 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerName="maas-api" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415933 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415940 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415954 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06ef4e8-7104-447b-9652-ac7dfce3b799" containerName="main" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.415960 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06ef4e8-7104-447b-9652-ac7dfce3b799" containerName="main" Jul 09 19:34:35.416013 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.416011 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f497d01-b6d9-4ffe-b3ab-1bc187433e7c" containerName="maas-api" Jul 09 19:34:35.416298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.416020 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.416027 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1360583-9297-4a41-a03a-4ccd391d0c6b" containerName="istio-proxy" Jul 09 19:34:35.416298 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.416033 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06ef4e8-7104-447b-9652-ac7dfce3b799" containerName="main" Jul 09 19:34:35.419018 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.419000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.421600 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.421580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"kube-root-ca.crt\"" Jul 09 19:34:35.421754 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.421675 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"openshift-service-ca.crt\"" Jul 09 19:34:35.421805 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.421798 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fz6n6\"/\"default-dockercfg-pl4vb\"" Jul 09 19:34:35.430366 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.430341 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf"] Jul 09 19:34:35.563323 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.563287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-podres\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.563592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.563335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-lib-modules\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.563592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.563467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-sys\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.563592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.563505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-proc\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.563592 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.563579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cq96\" (UniqueName: \"kubernetes.io/projected/317292b9-62f0-4799-9661-652bfc27741e-kube-api-access-9cq96\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664456 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-podres\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664456 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-lib-modules\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-sys\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-proc\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cq96\" (UniqueName: \"kubernetes.io/projected/317292b9-62f0-4799-9661-652bfc27741e-kube-api-access-9cq96\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-podres\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-proc\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664714 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-sys\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.664980 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.664711 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/317292b9-62f0-4799-9661-652bfc27741e-lib-modules\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.673188 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.673127 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cq96\" (UniqueName: \"kubernetes.io/projected/317292b9-62f0-4799-9661-652bfc27741e-kube-api-access-9cq96\") pod \"perf-node-gather-daemonset-6lttf\" (UID: \"317292b9-62f0-4799-9661-652bfc27741e\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.729277 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.729239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.856881 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.856851 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf"] Jul 09 19:34:35.858786 ip-10-0-131-3 kubenswrapper[2578]: W0709 19:34:35.858754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod317292b9_62f0_4799_9661_652bfc27741e.slice/crio-90958d39ae3af9f0c83b525f2d5b7e8a4afa8493bfde1e3c9459fdd96de28ca1 WatchSource:0}: Error finding container 90958d39ae3af9f0c83b525f2d5b7e8a4afa8493bfde1e3c9459fdd96de28ca1: Status 404 returned error can't find the container with id 90958d39ae3af9f0c83b525f2d5b7e8a4afa8493bfde1e3c9459fdd96de28ca1 Jul 09 19:34:35.978122 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.978088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" event={"ID":"317292b9-62f0-4799-9661-652bfc27741e","Type":"ContainerStarted","Data":"db4b644587e82255bb351baa9f82953add04b3388c13ef47f1ae789679d5a619"} Jul 09 19:34:35.978299 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.978144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" event={"ID":"317292b9-62f0-4799-9661-652bfc27741e","Type":"ContainerStarted","Data":"90958d39ae3af9f0c83b525f2d5b7e8a4afa8493bfde1e3c9459fdd96de28ca1"} Jul 09 19:34:35.978299 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.978171 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:35.996343 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:35.996291 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" podStartSLOduration=0.996275185 podStartE2EDuration="996.275185ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:35.994645474 +0000 UTC m=+2451.411359875" watchObservedRunningTime="2026-07-09 19:34:35.996275185 +0000 UTC m=+2451.412989595" Jul 09 19:34:39.325481 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:39.325452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ctn98_10d06887-6db9-405b-929f-fceb9b35ad05/dns/0.log" Jul 09 19:34:39.346561 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:39.346537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ctn98_10d06887-6db9-405b-929f-fceb9b35ad05/kube-rbac-proxy/0.log" Jul 09 19:34:39.418617 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:39.418591 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-79j55_daf59030-65db-468f-ba49-ee906a44d696/dns-node-resolver/0.log" Jul 09 19:34:40.108394 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:40.108371 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n9jq9_9559bce9-1971-40aa-8e5d-de6f89e2d607/node-ca/0.log" Jul 09 19:34:41.181273 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:41.181245 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689d8rk8d_761dce66-2b21-4b62-8aa7-ed63a5aeda95/istio-proxy/0.log" Jul 09 19:34:41.295584 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:41.295555 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-h2fmt_48c0afb1-990b-4773-a7f4-bfacd07d93d1/discovery/0.log" Jul 09 19:34:41.367808 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:41.367778 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/0.log" Jul 09 19:34:41.399030 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:41.399003 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/1.log" Jul 09 19:34:41.992280 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:41.992254 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-6lttf" Jul 09 19:34:42.247047 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:42.246977 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x9r5g_16fc8da6-cf69-4e42-8ba4-8ef43d4b4cdb/serve-healthcheck-canary/0.log" Jul 09 19:34:42.836311 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:42.836281 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8v9xg_0c4f64da-6aee-4c44-835b-bbfd5a80e428/kube-rbac-proxy/0.log" Jul 09 19:34:42.857277 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:42.857249 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8v9xg_0c4f64da-6aee-4c44-835b-bbfd5a80e428/exporter/0.log" Jul 09 19:34:42.882248 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:42.882224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8v9xg_0c4f64da-6aee-4c44-835b-bbfd5a80e428/extractor/0.log" Jul 09 19:34:46.075514 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.075480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-gm7r4_f66acc91-f60d-4884-91c5-a8f485d9a1f7/manager/0.log" Jul 09 19:34:46.094929 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.094904 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-gm7r4_f66acc91-f60d-4884-91c5-a8f485d9a1f7/copy-manifests/0.log" Jul 09 19:34:46.183244 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.183213 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689d8rk8d_761dce66-2b21-4b62-8aa7-ed63a5aeda95/istio-proxy/0.log" Jul 09 19:34:46.295169 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.295136 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-h2fmt_48c0afb1-990b-4773-a7f4-bfacd07d93d1/discovery/0.log" Jul 09 19:34:46.346645 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.346521 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/0.log" Jul 09 19:34:46.376556 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:46.376527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/1.log" Jul 09 19:34:47.449374 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.449345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9dfc7f7f4-dz7r9_8e6b3768-700a-4a08-86f9-65925793a372/manager/0.log" Jul 09 19:34:47.517773 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.517717 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qlfqx_cbce2148-883b-44e4-8ab2-63a9624bcc1f/openshift-lws-operator/0.log" Jul 09 19:34:47.739796 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.739707 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-554689689d8rk8d_761dce66-2b21-4b62-8aa7-ed63a5aeda95/istio-proxy/0.log" Jul 09 19:34:47.840165 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.840129 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-h2fmt_48c0afb1-990b-4773-a7f4-bfacd07d93d1/discovery/0.log" Jul 09 19:34:47.894454 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.894422 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/0.log" Jul 09 19:34:47.923933 ip-10-0-131-3 kubenswrapper[2578]: I0709 19:34:47.923904 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74949dcd89-kmrjg_5ad7303a-348c-43b4-be8f-f01aa6c912fb/istio-proxy/1.log"